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Preface 


Robotics and autonomous systems continues to be an area of increasing interest and investment worldwide. 
On a national basis, this is demonstrated through the recent capital investment from the UK government 
through the EPSRC in 2014, which resulted in the creation of eight dedicated centres/facilities across the 
country covering key areas of transport, healthcare, manufacturing, and unmanned systems to ensure that the 
UK will maintain its leading engineering and research capacity in robotics and autonomous systems. 

The 8 th Hamlyn Symposium on Medical Robotics was held at the Royal Geographical Society on 20 th to 23 rd 
June 2015. The theme for this year’s symposium was from whole body systems to micro-instruments’ with 
an impressive line-up of leading scientists and engineers in magnetic resonance (MR) and ultrasound 
imaging, image guidance, neurosurgery, and nanorobots, as some of our invited speakers. The keynote 
lecture was delivered by Professor Sylvain Martel, Polytechnique Montreal, who enlightened us on his 
experiences in using nanorobots and bacteria to fight cancer; Professor Andreas Melzer, from University of 
Dundee, spoke on recent developments in MR-image guided robotic-assisted interventions and focused 
ultrasound; Professor Nassir Navab of TUM, Germany and Johns Hopkins, shared his expertise and recent 
advances in robotics and control for patient specific imaging; Professor Garnette Sutherland, University of 
Calgary, outlined his experience with image guidance in neurosurgery, a new area of focus for the meeting 
this year. This year’s Storz-Hopkins lecture was delivered by Professor Alex Mottrie, who provided his 
insight and vision on the emerging technologies that might influence the future of robotic surgery. 

A total of 80 papers were submitted from 13 countries and 40 institutions, and after systematic peer review, 
47 papers were selected for presentation at the Symposium. The topics covered ranged from MR and 
Ultrasound-guided Interventions, Continuum Robots and Navigation, to Training and Microsurgery 
Platforms. 

We were delighted to see our workshop programmes continue to grow and flourish, held on the days pre- and 
post- the Symposium. Topics focused on some of the following areas: cognitive surgical robotics; MR- 
compatible robotics and interventions; paediatrics; imaging; flexible access surgery; microbots; additive 
manufacturing; and neurosurgery. 

As with last year, two of the workshops were organised in conjunction with the EPSRC-NIHR Healthcare 
Technology Co-operatives (HTC) Partnership Award on Devices for Surgery and Rehabilitation. In 
collaboration with the Trauma HTC at University Hospital Birmingham NHS Foundation Trust, the 
Wearable and Assistive Robots workshop had an impressive line-up of international leaders in the field. The 
Augmented Reality and Surgical Imaging Guidance Workshop was supported by the Enteric HTC at Barts 
Health NHS Trust and Queen Mary University of London. 

We were particularly excited to host the first Surgical Robot Challenge following on the Open Platforms for 
Medical Robots workshop at the 7 th Hamlyn Symposium, in partnership with Intuitive Surgical, Kuka 
Robotics, and Applied Dexterity. The purpose of the Challenge was to foster collaboration, share resources 
and tackle some of the unmet technical/clinical challenges in robotic surgery. A large number of submissions 
were received, with 17 teams bringing their kit to London to compete over a 3 -day competition, starting in 
the lab, followed by presentations to an esteemed judging panel and cumulating in the short-listed finalists 
presenting to the Hamlyn Symposium wider audience. 

A lively debate was hosted, based on an opinion piece entitled ‘Surgical Robotics: The Next 25 Years \ This 
highlighted that despite the impressive feats of innovation to date, there is a need to continuously improve 
surgical robotics through advances in precision medicine, personalised healthcare, and quality-of-life 
improvements. It was noted that future clinical attention will likely be paid to the development of smart, 
miniaturized, mechatronically enhanced or robotically assisted surgical instruments. Such smart instruments 
will be integrated with advanced imaging and sensing techniques, combined with instrumentation passed 
through the device for performing early diagnosis and interventions. 
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We would like to thank the International and Local Programme Committees, the Workshop Organising 
Committee and the Local Organising Committee for giving up their valuable time to ensure timely review of 
the submitted papers, to shape an excellent symposium programme. 

We are also grateful to the team who have worked behind the scenes and for their continuous effort in 
managing all aspects of the Symposium organisation. In particular, thanks to Valentina Vitiello, Su-Lin Lee, 
Raphaele Raupp, Robert Merrifield, Hawkeye King, Christos Bergeles, Benny Lo and Charence Wong. 

Lastly, special thanks go to Lady Hamlyn. This would not be possible without the generous philanthropic 
support from both the Helen Hamlyn Trust and Lady Hamlyn herself. 

It was our great pleasure to welcome attendees to the 8 th Hamlyn Symposium in London. 

June 2015, London 

Karen Kerr, Guang-Zhong Yang, Ara Darzi 
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INTRODUCTION 

Transcranial magnetic resonance-guided focused 
ultrasound (MRgFUS) involves the use of ultrasound 
energy to induce a bioeffect in a targeted area of the 
brain. An ongoing study at the Hospital for Sick 
Children proposes that intraventricular hemorrhagic 
(IVH) blood clots in premature infants can be treated 
with ultrasound-induced cavitation, and may result in 
better patient outcomes than current treatment methods. 
The procedure is performed inside a magnetic resonance 
imaging (MRI) machine. Precision positioning (±2 mm) 
of a focused ultrasound (FUS) transducer in five degrees 
of freedom (DOF) is required to perform the procedure 
within the contstraints of a neonatal incubator system. 

Melzer has developed a 5 DOF MRI compatible 
arm that has been retrofitted for FUS applications [1,2]. 
Yiallouras has developed a 3 DOF robot for FUS 
delivery to the prostate [3]. These robots lack the range 
of motion, payload capacity and/or dexterity required 
for transcranial FUS applications. 

This paper describes an original 5 DOF, MRI 
conditional robot that has been developed to manipulate 
a FUS transducer for transcranial therapies. 

MATERIALS AND METHODS 

MRI-compatible materials are used for robot 
construction. These include plastics (Delrin, Ultem, 
ABS, Polycarbonate), brass, aluminum and 316 
stainless steel for structural components. Custom 
designed parts were manufactured using CNC 
machining techniques. Nonmagnetic linear ball bearings 
(Del-Tron Precision Inc, CT, USA) are used as drive 
components for axes 1, 2 and 3 as shown in Figure 1. 
Glass ball bearings (Stock Drive Parts & Sterling 
Instruments, NY, USA) are used for rotational axes 4 
and 5. Nonmagnetic piezoelectric motors (USR60- 
E3NT, Shinsei Corp, Japan) are used to drive all five 
axes. The motors are compatible with the drive 
electronics from a clinical Phillips Sonalleve MRgFUS 
system, facilitating installation of the system into any 
MRI facility already equipped with a Phillips MRgFUS 
system. 

Robot components were tested for artifacts created 
during imaging. Individual robot components were 
placed immediately beside a cilindrical 4 L mineral oil 
phantom and imaged using T1 -weighted, T2- weighted 


and spoiled gradient echo based thermal map sequences. 
Signal to noise ratio (SNR) was measured. Motors were 
moved farther away in increments of 1 cm and imaging 
was repeated. This process was performed both with the 
motors in motion and with the motors stationary. 

A clinical FUS transducer (12cm focal length, 1.2 
MHz, ImaSonic, France) is attached to a water-filled 
coupling bag to provide a consistent transmission media 
of ultrasound energy into the head of the patient. This 
assembly weighs 1.41kg, and defines the payload 
requirement. 

Motion of the robot was tested under load to 
quantify payload capacity. Between 1 and 5, 1 kg brass 
weights were attached to the end effector of the robot in 
place of the FUS transducer. The robot was then moved 
through its full range of motions. 


System translates int o MRI 



Fig. 1 - Robot design with joints labeled. Utrasound coupler 
(blue) provides an acoustic path to the skull. 


Robot dexterous workspace was determined by 
calculating end effector position at each combination of 
joint limits. The smallest translational range of these 
values constitutes the dexterous workspace. The 
dexterous workspace for this application is defined as 
the volume the robot can target while maintaining a 
normal trajectory to the skull. An approximation of the 
brain volume of a 1 month old neonate was calculated 
based on a sphere of diameter equal to the major 
diameter taken from a computed tomography image 
(412 cm 3 ). Dexterous workspace results were compared 
to this worst-case volume approximation. 


1 



An inverse kinematic control scheme was 
implemented in MATLAB 2014b. A command in the 
form of an end effector pose (X, Y, Z, a, P) is entered 
into a console. The robot then moves to that pose. 

Spatial accuracy data was acquired with the use of a 
Polaris infrared tracking system (Northern Digital Inc, 
ON, Canada, ±0.25 mm accuracy). A reference fiducal 
was placed on the stationary base of the robot, and 
another fiducial was placed on the end-effector of the 
robot. Equation 1 was applied to determine the end- 
effector pose in the robot’s origin frame. Matrix T 
represents a 4x4 frame transformation, subscript NDI 
refers to the Polaris tracker origin, ee to the end effector, 
org to the robot’s origin frame, and ref to the reference 
fiducial. 


Figure 3). Reachable workspace extends well beyond 
skull volume. When placed in the center of the MRI 
bore, the robot workspace was not affected. 



Fig. 3 - Dextrous workspace (green), covers 37.4% brain 
volume, and provides significant access to IVH-relevant 
anatomy (ventricular system, blue). Typical entry trajectories 
shown in red. 


°' s T e e = ( Hr8 T ref )( NDI T re f)' l ( NDI T ee ) (1) 

The robot was commanded to move to 20 random 
points. The euclidean norm between the commanded 
position and the position measured by the Polaris 
tracker was calculated based on the average of 15 
measurement readings. This process was then repeated 
with the tracker incorporated into the robot positioning 
feedback loop to improve system accuracy. The robot 
was moved to the commanded position, a position 
measurement was taken, and the robot position 
corrected. This experiment was first performed outside 
of the MRI magnetic field, then again inside the field. 

The speed of each axis was determined by 
measuring the travel of each axis, commanding the 
robot to move through the full range of joint motion, 
and recording the time to complete the action. 


RESULTS 


The robot showed normal operation when placed 
inside the MRI bore indicating successful MRI- 
compatible construction. 



Fig. 2 - Image distortion created by motors on a 4 L MRI 
phantom at 0, 3 cm and 6 cm. 


Intrinsic system accuracy was measured to be 4.2 
±1.4 mm. When the Polaris tracker was used to close 
the feedback loop, system accuracy was 0.7 ±0.8 mm. 
Table 1 shows the joint characterization results. 


Table 1 - Robot joint characteristics 


Axis 

Type 

Travel 

(mm, 

deg) 

Resolution 
(pm, deg) 

Speed 

(nim/s, 

deg/s) 

ji 

P 

226 

1.3 

1.3 

J2 

P 

152 

2 

2.0 

J3 

P 

152 

1 

1.4 

J4 

R 

360 

8.18 x 10' 3 

8.2 

J5 

R 

90 

8.18 x 10' 3 

8.2 


DISCUSSION 

We have successfully designed and manufactured a 
5 DOF, MRI-conditional robot capable of manipulating 
a FUS transducer for transcranial FUS delivery. The 
accuracy, dexterity, workspace, payload capacity and 
speed of the system have been shown to meet design 
requirements. 

Future work will focus on additional system 
accuracy and repeatability characterization inside the 
MRI bore, improving the system control model, 
improving system accuracy, phantom and animal testing 
of robot controlled FUS sonications and continuing to 
improve integration of the control interface with the 
MRI console. 


Distortions in the MR image due to large metal 
objects (such as the drive motors) were found to be 
negligible at distances longer 6 cm from the imaging 
field. Worst-case SNR recorded was 54% degredation 
(T2 weighted, motors in motion) compared to baseline. 
Figure 2 shows distortion as a result of the motors at 
varying distances. 

The robot successfully moved through its full range 
of operation with a maximum payload of 5 kg. 

Dexterous workspace volume was found to be a 
rectangular prism of dimensions 4.9 x 4.1 x 17.6 cm, 
covering 37.4% of the neonate skull volume, including 
significant coverage of the ventricular system (shown in 
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INTRODUCTION 

Percutaneous intervention of liver tumours conducted 
under Magnetic Resonance Imaging (MRI) guidance 
offers better soft tissue contrast compared to the 
conventional ultrasound guidance [1]. While open MRI 
scanners have been employed for image-guided needle 
insertions in laser ablation (LA) procedures, their lower 
field strength limits image resolution and increases scan 
times [2]. Closed -bore scanners are more common; 
however the access to the patient is severely restricted. 
Consequently, image-guided needle insertions require 
repeatedly moving the patient bed in and out of the 
scanner bore, resulting in lengthy procedures. This 
paper introduces a new needle-guiding robot that 
operates in closed-bore MRI scanners in order to reduce 
procedure duration and increase targeting accuracy. The 
advantages of robot-assisted needle insertion are 
highlighted with a comparative phantom study. 

MATERIALS AND METHODS 

The needle-guiding robot builds upon our proof-of- 
concept [3] while introducing important new features. 
Firstly, a new design with four degree-of-freedom 
(DOF) results in a larger workspace and allows double 
inclined needle insertions, which are performed 
manually outside the bore (Fig. 1). Secondly, the robot 
registration in the MRI scanner is conducted using 
passive markers and an image processing method [4]. 
This approach does not require special imaging 
sequences and is therefore readily usable on different 
MRI scanners. Differently from other robots for liver 
intervention [5], [6], the prototype is designed for 
closed-bore scanners and employs pneumatic actuation 
which has negligible effects on the MR image quality 
[3]. This prototype is intended to operate within the 
established MRI-guided LA workflow, in which 
mechanical ventilation is briefly paused during needle 
insertion in order to minimize liver motion [7]. 

RESULTS 

The potential benefits of the needle-guiding robot were 
evaluated with a comparative phantom study in a 3T 
closed-bore MRI scanner (GE, Discovery, MR750). The 
study employed a gelatine phantom containing four 
Vitamin-E capsules (Boots, UK), representing the target 
lesions. The phantom was positioned inside a plastic rib 
cage model, at the scanner isocenter. Silicone tape was 


attached to the ribs to make them visible on the MR 
images (Fig. 2). In the first part of the study, a coaxial 
needle (Somatex, Germany) was manually inserted in 
the phantom, outside the bore, based on the MR images. 
A grid of equally spaced cod-liver-oil capsules (Boots, 
UK) was positioned above the rib cage to provide a 
visual reference. Four sets of insertions were performed 
by two radiologist registrars aiming at different targets. 
Initial MR images (3Plane Loc SSFSE, TR = 1662.7 
ms, TE = 78.9 ms, FA = 90°, FOV = 480 mm x 480 
mm, Slice Thickness = 8 mm, Slice number = 65) were 
acquired to locate the lesions with respect to the capsule 
grid and to plan the needle insertion between the ribs. 



The position of the inserted needle was confirmed with 
a verification scan: the insertion was repeated until the 
needle artefact intersected with the lesion (Fig. 3). The 
number of insertion attempts and the time elapsed from 
the start of the planning scan to the end of the 
verification scan is reported for each target in Table I. 



Fig. 2 Axial MR image of the phantom. 
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Fig. 3 MR images of a successful needle insertion. 


The second part of the test involved using the needle - 
guiding robot to plan and assist the needle insertion. The 
robot was set up next to the plastic rib cage (Fig. 4), and 
automatically registered in the scanner coordinate 
system. Finally, a specially designed graphical user 
interface was employed to plan the needle insertion, 
which was performed manually outside the bore with 
the assistance of the needle guide. Although the robot 
setup took 30 minutes, it could be completed well 
before the clinical procedure and would therefore not 
affect its duration. The robot registration was completed 
in 5 minutes, which would be additional to the 
conventional LA procedure. Differently from the 
manual tests, all four targets were successfully 
punctured at the first attempt (Table II). 


Table I Manual test 


Target 

Axial 

Sagittal 

Insertion 

Duration 

angle q 3 

angle q 4 

attempts 

(minutes) 

1 

0° 

30.0° 

2 

27 

2 

0° 

28.3° 

2 

29 

3 

36.9° 

21.5° 

3 

26 

4 

19.6° 

-11.7° 

4 

35 


Table II Robot-assisted test 


Target 

Axial 

Sagittal 

Insertion 

Duration 

angle q 3 

angle q 4 

attempts 

(minutes) 

1 

-5° 

7.9° 

1 

9 

2 

13.5° 

-11.4° 

1 

12 

3 

-7.1° 

-14.3° 

1 

15 

4 

Tj 

oo 

o 

17.2° 

1 

10 


DISCUSSION 

The results of the phantom study show that the needle- 
guiding robot allows targeting lesions of clinically 
representative size [2] in a shorter time compared to 
conventional manual insertions. In particular, in the 
robot-assisted test all targets were successfully 
punctured at the first attempt. This aspect could be 
particularly beneficial in clinical procedures since 
reducing the number of insertions also diminishes the 
risk of damaging blood vessels and healthy tissue, thus 
enhancing patient safety. Finally, the manual test 
showed that double inclined insertions are substantially 
more complex since they require considerable amount 
of training and 2D to 3D extrapolation skills. This 
suggests that the needle-guiding robot could be 
particularly beneficial for less experienced clinicians, 


allowing them to independently conduct MRI-guided 
LA procedures. Future work will include further tests 
with more experienced clinicians in preparation for the 
clinical trials. 



Fig. 4 CAD model of the robot setup in the MRI scanner. 
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INTRODUCTION 

The goal of targeted delivery systems for therapeutic 
agents, e.g, drugs, nucleic acids and cells, is to 
overcome the limitations of conventional therapeutic 
strategies by avoiding the side effects of systemic 
administration. Most delivery systems under 
investigation, however, such as magnetic nanoparticles 
and smart triggerable materials, are affected by low 
controllability and the impossibility of retrieving the 
delivery system once the therapy has been completed, 
thus raising toxicity- associated risks [1]. Medical 
robotics and microsystem technologies can potentially 
contribute to the development of devices able to 
navigate a wide network of small -diameter canals and to 
controllably get in the hard-to-reach areas of the human 
body [2]. Several strategies have been proposed for 
microrobot locomotion and magnetic propulsion has 
emerged as one of the most promising approaches [3]. 
This paper deals with the design, prototyping and 
preliminary testing of a magnetically actuated robot able 
to navigate through relatively small diameter body 
canals (spine, urinary system, ovary, etc.), and to 
perform in- situ release of therapeutic agents. A bi- 
component system was developed, consisting of a 
carrier, in which the therapeutics are embedded, and a 
piston. External magnetic fields are used to propel and 
independently bring the robot components to the 
neighborhood of the target site, whereas intermagnetic 
attraction forces, acting when the carrier and the piston 
are in close proximity, are responsible for the docking 
between the parts that compress a drug-loaded hydrogel, 
thus activating drug release. 

MATERIALS AND METHODS 

The system consists of two millimeter- sized near- 
neutrally buoyant components ( i.e . carrier and piston) 
both containing a spherical permanent magnet (NdFeB 
N45, 1 mm in diameter) (Fig. 1). The carrier consists of 
a polydimethylsiloxane shell (PDMS, SYLGARD® 184, 
Dow Corning) provided with holes to allow drug release 
and with NdFeB powder (Magnequench MQA-37-11, 
MolyCorp) embedded in its tip to increase magnetic 
heterogeneity thus enhancing independent control of the 
robot components [4]. Navigation of robot components 
can be performed by using an MRI scanner, a coil 
system or permanent magnets. The therapeutic strategy 
is articulated as following: i) the carrier is injected into 
the desired body lumen and navigated to the target site 
where it keeps its position; ii) the piston is subsequently 


injected, oriented and brought to the target area as well. 
Thanks to the geometric and magnetic differences 
between the piston and the carrier, independent control 
can be accomplished [4] ; iii) when the distance between 
the two components falls below a critical threshold, the 
magnetic attraction between the carrier and the piston 
prevails over the external control force. Docking, 
compression of a hydrogel and drug release, occur; iv) 
at the end of the procedure the bi-component robot is 
magnetically retrieved and extracted from the body. 



Fig. 1 Bi-component robot design and elements. 

In the design of the millirobotic system, the selection of 
the drug-embedding material is challenging. Hydrogels 
based on agar and gelatin were selected for their 
biocompatibility and for the possibility to infuse them 
not only with drugs, but also with nucleic acids or cells, 
thus accommodating future therapeutic treatments. 
Different hydrogel compositions, with different agar 
concentration (4%, 2%, 1%, 0.75%, 0.5% and 0.25% 
w/w agar concentration with gelatin concentration fixed 
at 1% w/w) were fabricated and mechanically tested. In 
order to identify the best hydrogel composition, finite 
element model (FEM) simulations were carried out 
using Abaqus and MATLAB®. The aim was to calculate 
the deformation of the hydrogel cylinder due to the 
force exerted by the piston, which corresponds to the 
dipole-dipole interaction force between the embedded 
permanent magnets. 

An anticancer drug (doxorubicin) commonly exploited 
for the treatment of different kinds of cancer was 
embedded into the hydrogel matrix at a concentration of 
200 pg/ml. Piston and carrier components were 
fabricated in PDMS (monomer- curing agent ratio 10:1) 
by exploiting custom made 3D printed molds and a 
dedicated setup to orient and accumulate NdFeB 
powder on the carrier’s tip. The polymeric shell 
components were assembled by embedding the 
permanent magnets; then, 10 pL of “hydrogel+drug” 
solution were injected into the carrier and let solidify in 
situ (Fig. 2). 

Finally, the correct operation of the proposed platform 
was evaluated through magnetic locomotion, docking 
and drug release tests. The amount of doxorubicin 
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released by different prototypes (ft=3) robot was 
assessed quantitatively through spectrophotometric 
analysis. In vitro tests were carried out by evaluating the 
effects that the devised procedure produced on human 
bladder cancer cells (T24, ATCC®, HTB-4™), both in 
the docked and undocked configurations. 



Fig. 2 Robot prototypes: A) carrier with doxorubicin- 
embedded hydrogel; B) piston. 

RESULTS 

At the initiation of hydrogel compression, the dipole- 
dipole attraction force between the embedded 
permanent magnets is 1.3 mN. FEM simulations 
revealed that, due to this compression force, a 0.25% 
agar - 1% gelatin hydrogel produced the best results, 
undergoing a deformation of about 38%. Further 
simulations and tests revealed that the embedded 
doxorubicin does not negatively affect hydrogel 
behavior, thus leading to the choice of this formulation 
as drug-loaded matrix. The locomotion of the robot 
components and the correct operation of the docking 
procedure were demonstrated in a liquid environment 
by exploiting a permanent magnets -based control (Fig. 
3). 


The carrier is dragged to the target position and the piston is injected in 
the working channel 



The piston is independently brought towards the target while the carrier 
keeps its position 



When the carrier and the piston are in close proximity, intermagnetic 
attraction force prevails and docking occurs 





Robotic system in the 

[ 

docked configuration 


Fig. 3 Experimental validation of navigation and docking. 

The efficacy of the drug release procedure was 
evaluated through spectrophotometric analyses, by 
measuring the amount of doxorubicin released by each 
robot prototype after docking. Results demonstrated that 
the spontaneous release without docking (control 
samples) is negligible, whereas 6.3±2.4 ng of drug were 
released by each robot prototype due to docking (Fig. 
4A). Procedure efficacy was furtherly demonstrated by 
treating T24 cells with docking and undocking robots. 
In vitro tests revealed that the amount of doxorubicin 


released by the carrier is able to produce the desired 
toxic effect on cancer cells. Both reactive oxygen 
species (ROS) analysis, carried out 24 h after the 
docking procedure (fig. 4B), and DNA measurements, 
carried out 72 h after system testing, confirmed that the 
docking mechanism is highly effective to activate the 
drug release, thus damaging cancer cells. 



Fig. 4 Targeted therapy system validation. A) Doxorubicin 
temporal release profile in the docked (squeezed robot) and 
undocked configuration (control). Measures were performed 
in triplicate: average curves are reported; B) fluorescence 
images for the different sample types, 24 h after the treatment 
(nuclei in blue, ROS production in red). Negative control = no 
treatment; positive control =10 pg/mL doxorubicin directly 
dispersed in the culture medium. Positive control is similar to 
the “docked robot” condition. Negative control is close to the 
“undocked robot” condition. 

DISCUSSION 

This paper reports a novel millirobot concept, enabling 
reliable targeted therapy in body regions currently 
reachable only by means of systemic drug 
administration or low-dexterity instruments. Magnetic 
locomotion and release tests were carried out, thus 
demonstrating that the proposed release mechanism, 
based on controlled squeezing of a drug-loaded matrix, 
is reliable and efficient and is able to produce a 
therapeutic effect on cancer cells. 
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INTRODUCTION 

Robot-assisted laparoscopic radical prostatectomy 
(RALRP) using the da Vinci surgical system is the gold 
standard treatment option for organ confined prostate 
cancer. However, the rates of positive surgical margins 
(cancer left behind after surgery) still range between 9% 
and 30%, depending on the center [1]. The main reason 
why the best trade-off between achieving oncological 
success (cancer removal) and functional success 
(continence and potency) cannot be achieved is the 
inability to localize, intraopratively, the location and 
extent of cancer. Advances in MRI may provide 
spatially localized information to fill this void and aid 
surgical planning. 

In this work, we present a novel MRI-Ultrasound 
guidance system for the da Vinci which involves a 
novel intraoperative segmentation-based MR- 
Ultrasound deformable registration method that is 
integrated into a clinically used robotic trans-rectal 
ultrasound (TRUS) imaging system [2], which in turn 
can be registered to the da Vinci system's coordinate 
frame. Using such a system, a 3D MR volume and the 
preoperative surgical plan can be mapped to the da 
Vinci system’s coordinate frame. The surgical 
instrument can then be visualized, in real-time, with 
respect to the preoperative deformed MR volume. In 
addition, since the TRUS imaging system is robotic, it 
can track the tip of a da Vinci surgical instrument 


automatically, and is registered to the preoperative MR 
volume, the surgical instrument itself can be used as an 
intuitive and easy to use control device for the surgeon 
to manipulate both MR and TRUS images in real-time 
during the procedure. After initial testing and validation 
on a prostate phantom and a data-set of n=6 patients 
offline, the system was used with a clinical da Vinci 
surgical system inside a robotic operating room and 
tested on two patients undergoing RALRP. We outline 
the system architecture and present our experience with 
the system in the first two patients. The feasibility and 
application of fused MRI and TRUS during radical 
prostatectomy has not been demonstrated before. Given 
the significant rates of positive surgical margins still 
reported in the literature, the presented system has 
potentially significant clinical benefits. 

MATERIALS AND METHODS 

The components of our image-guided system are 
illustrated in Figure 1. The system comprises: (i) an 
ultrasound system with a motorized TRUS transducer 
mounted on a brachy therapy setup, (ii) an external PC 
for image registration and display on the da Vinci 
console, and (iii) registration and tracking software. A 
previously designed and clinically used robotic TRUS 
manipulator was used for automatic and remote rotation 
of the TRUS transducer during the procedure [2]. 




Fig. 1 Schematic figure of the system components. Dotted black arrows show the transformation between various coordinate systems 
involved in this system: da Vinci (dV), robotic TRUS (RT), ultrasound sagittal image (USs), ultrasound transverse image (USt) and 
MR volume (MR) . 
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Imaging interface inside the daVinci®S 
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Fig. 2 (a) Real-time imaging interface using the da Vinci S TilePro feature and our custom MR imaging GUI. (b) An example image 
of the system in clinical use and imaging workflow. Both deformed MR and real-time TRUS are displayed to the surgeon. Post- 
operative whole-mount histopathology imaging is used to validate the cancer localization performance using MRI. 


The sagittal TRUS imaging plane is automatically 
repositioned using the robot, so that the 2D ultrasound 
image continuously contains the tip of a specified da 
Vinci surgical manipulator. Automatic instrument 
tracking is achieved by means of a rapid, accurate and 
clinically feasible intraoperative registration technique 
that solves for the rigid homogeneous transformation 
between the da Vinci and the TRUS robot coordinate 
systems (Figure 1) [2]. 

The prostate gland on each transverse slice in the 
preoperative T2-weighted MR volume is segmented 
manually by a radiologist before surgery. After 
interpolating the intraoperative 3D TRUS B-mode 
images into a 3D grid in order to obtain transverse slices 
from the sagittal volume, we employed a real-time 
semi-automatic algorithm for 3-D segmentation of the 
prostate in the ultrasound volume. Based on the 
segmented surfaces of the prostate in the TRUS and MR 
volumes, we construct binary volumes. The two binary 
volumes can then be registered to each other in order to 
obtain a displacement map which can be applied to the 
MR volume. First, the MR binary volume is rigidly 
aligned (and scaled) to the TRUS binary volume using 
the principal axes transformation. Next, we deform the 
aligned MR volume to match the TRUS volume. The 
registration algorithm, based on the variational 
framework, minimizes the sum of squared differences 
between the two binary volumes with an elastic 
regularization of the displacement map. 

RESULTS AND DISCUSSION 

Two patients (ages 55 and 72; prostate specific antigen 
13.5 and 28.5 ng/ml) with clinically organ confined 
prostate cancer undergoing RALRP agreed to 
participate in this institutional review board approved 
study. The imaging system interface and an example use 
case for intraoperative tumour localization is shown in 
Figure 2. For each case, after intraoperative TRUS to da 
Vinci and MR-TRUS registrations and activation of 


automatic tracking, the surgeon could examine the 
prostate anatomy and tumour locations by moving the 
registered surgical instrument around, placing it on the 
tissue surface in the area of interest and localizing the 
instrument tip with respect to anatomy seen in real-time 
sagittal TRUS images. The corresponding transverse 
MR slice is also displayed based on the position of the 
tool tip (shown in Figure 2). TRUS to da Vinci 
registration was performed successfully for both cases 
in less than 120 seconds and accuracy of less 
than 2 mm. MR-TRUS registration was performed for 
both cases in a mean time of 100 second and 
registration accuracy of 2.1 ± 0.3 mm and 2.5 ± 
0.1 mm for first and second case respectively. 

The performed surgical procedures were the first in 
which a surgeon was able to see registered MR images 
intraoperatively, and to refine the surgical planes 
accordingly for achieving a better surgical margin and 
functional outcome. The intraoperative MR imaging 
allows surgeons to fine tune the procedure based on 
anatomy. Without MRI, they would just have a certain 
number and proportion of cores involved on one side or 
the other, but no other information on location - hence, 
relying mainly on intuition. Given the controversy on 
the “best” surgical approach in RALRP, as reflected in 
the literature, this MR-guidance could be a powerful 
tool to trade-off positive surgical margins against 
potency. 
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INTRODUCTION 

Non-aneurysmal intracerebral hemorrhage (ICH) is a 
significant burden to quality of life, society, and life 
itself. It comprises approximately 20% of all strokes 
with 40% and 55% median mortality rates at 30 days 
and 1 year, respectively [1]. The current standard of 
care consists of conservative medical management and 
treatment of any risk factors for repeat hemorrhage or 
progression (e.g. hypertension, coagulopathy). Many 
studies have addressed the role of surgery in 
intracerebral hemorrhage [1]. There appears to be a 
significant benefit in highly selected patients, especially 
those in which the hematoma comes close to the surface 
and is in the supratentorial compartment. Additional 
injury to the brain might occur in reaching the 
hemorrhage surgically as they are frequently in eloquent 
regions. 

The ability to reach these hematomas through a 
trajectory that minimizes disruption of healthy brain and 
also allows for a dexterous aspiration within the 
hemorrhage is desirable. This may allow for 
decompression or removal of the hematoma and 
possibly salvaging the penumbra of at risk brain 
surrounding the lesion. A promising enabling 
technology are smaller, yet dexterous surgical 
manipulators and robots [2]. Tubular manipulators, also 
known as active cannulas [3], are particularly well 
suited for ICH surgery [4]. In this system, the needle- 
sized manipulator reaches the hemorrhage on a straight 
path. The curved inner tube allows debulking of the 
accumulated blood from within by coordinated motion 
during aspiration. By using differently curved aspiration 
tubes subsequently, the majority of the hemorrhage can 
be evacuated in a minimally invasive manner. Based on 
our initial results presented in [4], we introduce two new 
and less computationally intensive optimization 
methods to select aspiration tubes maximizing the 
volume coverage. 

MATERIALS AND METHODS 

The tubular aspiration robot is composed of a stiff guide 
tube, which is straight in order to reach the hemorrhage. 
An aspiration tube, which is superelastic (NiTi) with 
diameter 1mm, is inserted through the guide tube. It is 
composed of a straight section followed by a precurved 
section with constant curvature (see Fig. 1). The robot 
has 3 degrees of freedom (DOF): translation of the 
guide tube, translation and axial rotation of the 


aspiration tube. The kinematics of this robot has been 
described in [4] and allows computation of the shape of 
the robot for a given set of actuator values q=( a,p b p 2 ). 

In this paper, we consider preoperative coverage 
planning. This planning problem is performed on 3D 
surface data representing the hemorrhage of the patient, 
obtained from computed tomography images and a 
straight entry path from the skull to the hemorrhage 
defined by the surgeon. We have shown that subsequent 
use of aspiration tubes with different curvatures leads to 
good coverage of the hemorrhage volume [4]. Thus, the 
optimal selection is performed from a set of potential 
tubes parameterized by the radius of the curved section 
S = {r lf ... , r n ] and L c = rn (Fig 1). The goal of optimal 
coverage planning is to find a subset S* E S with k 
aspiration tubes, which maximize the volume coverage 
of the hemorrhage, k can be predefined or incremented 
during planning until the clinically desired coverage is 
achieved. We quantify the covered volume as the ratio 
between the number of voxels (unit volume 1mm 3 ) 
reachable with the aspiration tube’s tip and the overall 
number of voxels representing the hemorrhage (see [4]). 
We previously presented a brute force algorithm [4], 
which determines the coverage for all possible 
combinations of k tubes in 5 and chooses S* as the 
combination with maximum coverage. For large k and 
n this combinatorial optimization problem increases in 
complexity by 0(n ) = n! (/c! ( n — /c) !) -1 . Thus, we 
propose two alternative algorithms in this paper. 

The first is a greedy algorithm, which initially selects r t 
with the best coverage from 5. In the second step, 5 is 
reduced to n — 1 elements and r 2 with best coverage is 
selected out of 5 and so on. Thus, the number of 
combinations is reduced compared to the brute force 



Fig. 1 Tubular aspiration robot with 3 DOF (a, p 1? p 2 ). 
Design parameters are tube lengths (L sl , L s2 , L c , and r). 
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algorithm with complexity 0(ri) = Ef=n-(k-i) i • For 
k = 1 the greedy algorithm achieves the same result as 
the brute force method. 

Our second algorithm approximates the optimal tube 
radii based on our empirical results obtained with the 
brute force algorithm. This approximation algorithm 
does not involve computation of coverage during the 
selection as it uses a heuristic to determine the 
aspiration tubes radii. Here, we select the first tube with 
radius closest to half of the hemorrhage’s extent in 
direction of the entry path. The second and third tube 
will be selected with smaller radii than the first 
aspiration tube (decremented in 1mm steps). The 
computational complexity is reduced to 0(n) = 1. 

We evaluated the performance of all 3 tube optimization 
algorithms on 8 representative patient ICH cases. Fig. 2 
illustrates the 3D surface models of the hemorrhages. 
For each case we randomly generated 5 straight entry 
paths from a random position on the skull surface to the 
centroid of the hemorrhage volume. Thus, we obtained a 
total of 40 trials. For each trial, we determined the 
coverage achievable for an initial set of tubes 5 = 
{6,7,8,9,10,11.12,13,14,15,16}, L sl =300, L s2 =140 (in 
mm), and for k = {1,2,3} using each optimization 
algorithm. 



Fig. 2 Intracerebral hemorrhage geometries for 8 patient cases. 

RESULTS 

The coverage distribution over all 40 trials for the brute 
force, greedy, and approximation algorithms are 
summarized in histograms in Fig. 3-5 for 1, 2, and 3 
tubes respectively. For all 40 trials, a volume coverage 
over 50% can be reached, which is considered clinically 
beneficial. It can be observed, that the concentration 
shifts significantly from 70-80% coverage with 1 tube to 
80-90% with 2 tubes. The usage of 3 aspiration tubes 
allows for coverage mostly over 80%. The greedy 
algorithm leads to comparable results to the brute force 
method. 
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Fig. 3 Coverage distribution for k = 1 aspiration tube. 



Fig. 4 Coverage distribution for k = 2 aspiration tubes. 



Fig. 5 Coverage distribution for k = 3 aspiration tubes. 

In terms of computation time, brute force takes about 10 
min, greedy about 22 s, and approximation algorithm 
about 6 s for one case on a regular PC (Matlab). 

DISCUSSION 

In this paper, we introduced two alternative 
optimization algorithms to determine a set of aspiration 
tubes, which maximizes coverage for evacuation of 
intracerebral hemorrhages. Our results indicate that 1 to 
3 tubes selected from an overall set of 11 precurved 
aspiration tubes result in clinically significant coverage. 
The approximation algorithm may be favorable in those 
clinical cases with no time for preoperative planning. 

Our current work focuses on computing an optimal 
motion plan (i.e. sequence of actuator values) for the 
selected aspiration tubes which allows for quick and 
safe evacuation of the hemorrhage in vivo. Extensive 
animal trials will be needed in the future to show 
clinical feasibility. Ultimately, we expect improved 
surgical outcome for patients suffering from 
intracerebral hemorrhages. 
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INTRODUCTION 

Steerable catheter ablation therapies have been 
investigated for effective minimally invasive treatment of 
cardiac arrhythmia. Minimally invasive catheter ablation 
procedure promises attractive advantages, such as 
minimal incisions, quicker recovery time, less bleeding, 
and other economic effects However, the complex and 
delicate nature of the procedure makes it difficult to steer 
the catheter manually, causing an extension of surgery 
time. Thus, well-trained and skilled surgeons are required 
to treat a large number of patients in an effective manner. 
Moreover, extended procedure times increases the 
radiation burden of both surgeon and patient. In order to 
solve these problems, a tele steerable catheterization 
system has been developed. Currently, the proposed and 
improved robotized steerable catheter system has 
capabilities to provide tension sensing for safe steering 
and feedback of catheter position/force control, catheter- 
tip force estimation based on shape analysis, robotic 
image guidance for less radiation exposure, and tele- 
steering for easier steering. 

MATERIALS AND METHODS 

The catheter system is illustrated in Fig. 1. Four linear 
guides were installed to control four tendons, and these 
were controlled with a motion controller and user 
interface. The catheter was tracked using magnetic 
resonance image guidance. Also, shape analysis based the 
catheter-tip contact force estimation had been developed. 
In addition, the tension was measured using a custom- 
developed tension sensor with a strain gauge. This system 
was designed to allow the operator to operate away from 
patient, the proposed robotized steerable catheter system 
provides: 1) the actuator system which reacts to the 
measured tendon tension and input from operator, 2) 
contact-tip contact force estimation, and 3) 3D catheter 
magnetic tracking combined with image guidance. 

Catheter tip 

The MR-compatible catheter tip was made of twelve 
segments connected together. Each segment included a 
helical structure bended 4-way using four tendons, and 
the catheter-tip satisfied required dimensions used for 
cardiac catheterization, such as, the diameter was within 
3mm; inner lumen diameter was 1.7mm to allow insertion 


of optional elements and irrigation. Each tendon guiding 
channel was 250pm in diameter, and the total length of 
the catheter tip is 112mm. In addition, a linear stage 
actuation was employed to provide axial movement of the 
catheter-tip, and also variable deflection curvature radius 
was achieved through sliding of a nitinol tube into the 
catheter tip. Thus, the catheter-tip provided a full 3D 
workspace. 



Tension Sensor 


Steerable Catheter 


Magnetic Tracking 


Linear Actuation 


Fig. 1 Schematic diagram showing configuration of the 
proposed robotized tele catheterization system. 


Tension sensing 

The shape of the catheter tip is dictated by tension on the 
tendons. The tension is an essential parameter for safe and 
efficient steering, and preventing slack on the tendons. 
Additionally, during tele-control, the mechanical safety 
conditions are regulated with tendon tension 
measurements. 


To this end, a simple cantilever beam structured tension 
sensor was developed as shown in Fig 2. Taking the linear 
stage of the beam deflection, the tension sensor had initial 
bending radius. A strain gauge attached to the middle 
surface of the cantilever beam measured the voltage 
difference depending on the applied tension. The tension 
sensor was calibrated, and it showed an accuracy of 
96.5%. 
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Fig. 2 The simple cantilever beam structured tension sensor, and 
tension sensor calibration result. 


Force Estimation 

The importance of knowing the catheter-tip contact force 
is important due to risk of perforation during catheter 
insertion and this has motivated the development of 
numerous catheter-tip force sensors. However, many 
existing contact force designs cannot be easily integrated 
[1]. Moreover, existing force sensing devices employ 
fiber-optics, and strain gauge methods have difficulty in 
operating under small temperature changes. Therefore, we 
proposed a new contact force and angle estimation 
method based on the Cosserat rod theory which is suitable 
for the large non-linear deflection of a flexible body. To 
achieve real-time and analytical force estimation, we 
simplified the original Cosserat rod model and made 
several reasonable assumptions for solving it analytically. 
The force estimation was evaluated using several different 
scenarios such as the contact force due to internal tension 
and external contact as shown in Fig. 2. During these 
experiments, the configuration of catheter was extracted 
image processing. These result of the contact force 
calculation showed promise: The accuracy in estimating 
force magnitude was 89.86%, and for the force direction 
was 88.13%. 



Segments Tracking 




(a) (b) 

Fig. 3 Results of the contact- tip force estimation: (a) Due to 
internal tension, and (b) Due to external contact. 



Fig. 4 3D and 2D Robotic Image Guidance. 


Robotic Image Guidance 

The image guidance platform consisted of the 3D CT 
reconstruction of an experimental phantom glass heart 
and its four chambers. Additionally, a 2D planar 
representation of one of its chambers (i.e. the left atrium) 
was also available from 3D to 2D flattening algorithm [2]. 
These planar representations provided a flattened map of 
the complex chamber anatomy on which the operator 
selected catheter targets for ablation. A commercial 
magnetic coil tracking system (NDI Aurora® EM) with a 
5 -DOF coil was used to measure the 3D position of the 
catheter tip in this experimental phantom glass heart 
setup. 

FUTURE WORK & CONCLUSIONS 

Tension sensing, catheter-tip contact force estimation and 
robotic image guidance have been developed, and we 
envisage that the fully-integrated proposed system has 
potential for safer and cost-effective left atrial chamber 
tele-catheterization procedures. Therefore, in the next step 
of work, the tele-catheterization system will be examined 
clinically, and compared with manual catheterization. 


Additionally, capabilities of the tele catheterization 
system satisfy necessary requirements of the proposed 
position/ force control algorithm described in [3, 4]. 
Therefore, future work will exploit these capabilities to 
improve the control of steering and of the interaction 
between heart tissue and catheter tip for robot-assisted 
catheter ablation procedures. 
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INTRODUCTION 

Concentric tube robots (see Fig.l) are continuum robots 
that exhibit shape change through the insertion and 
rotation of pre-curved super-elastic tubes. Accurate 
control of the tip of a concentric tube robot requires 
thorough mechanics modelling that accounts for the 
individual tube parameters and their interaction. While 
extensive research has been performed on their highly 
non-linear modelling, no methodologies account for 
uncertainty in the parameters and actuation mechanism. 
This paper explores the application of neural networks 
for identification of the non-linear model of particular 
concentric tube robot mechanisms. 

MATERIALS AND METHODS 

Even though the research community has converged 
with regards to concentric tube robot mechanics 
modelling [1][2], all proposed methods rely on the 
underlying model of the robot and are susceptible to 
erroneous estimation of the multitude of parameters that 
govern the robot’s behaviour, i.e. the stiffness, 
curvature, length, and Young's Modulus of individual 
tubes. Hence, a method to estimate the kinematics and 
inverse kinematics in situ , i.e. for a given robot but 
without requiring knowledge of its parameters, would 
be beneficial. 

The approach explored here is based on machine 
learning, and, more specifically, neural networks. Until 
now, the demonstrations of neural networks for 
continuum manipulators is the neural network controller 
of [3], and limited work, [4], has explored inverse 
kinematics using neural networks. The current paper 
demonstrates that neural networks may be viable 
alternative to model-based kinematics approaches. 
Multi-layer perceptrons (MLP) are used as the 
architecture of neural networks (NN). The basic 
building block is the perceptron, i.e. a multiple input 
and single output neuron (see Fig. 2(a)). The perceptron 
is characterised by the weights, Wi , of its inputs, pi , a 
comparison element, b, and its activation function F : x 
E R — > [-1, +1]. Sets of N perceptrons are cascaded in 
hidden layers to form MLP to map from R k — > R m , 
where k is the number of inputs and m is the number of 
outputs (see Fig. 2(b)). 

The NN should possess the appropriate activation 
function F, together with the minimum number of 
neurons n and hidden layers N, to map inputs x E R n to 
outputs in y E R m with the required precision. 


Using Matlab, these parameters are explored for the 
kinematics and inverse kinematics for a concentric tube 
robot comprising 3 tubes. Two tubes form a variable- 
curvature section, i.e. two tubes of identical stiffness, 
length, and pre-curvature that rotate independently to 
control the section’s curvature but translate 
simultaneously, and the third tube of fixed pre- 
curvature. This robot possesses a total of 5 degrees-of- 
freedom - 3 in position and 2 in pointing orientation - 
and its parameters are given in the Table. Each tube has 
D-times larger stiffness k than its directly inner tube, 
thus following the “dominant stiffness” rule of [1]. The 
robot, illustrated in Fig. 1 is stable in its workspace. 



Fig. 1 A concentric tube robot with a variable-curvature 
section (2 tubes) and a fixed-curvature section (1 tube). Tube 
extensions and rotations control the tip. Angles a correspond 
to twist angles that vary along the robot’s shape to account for 
robot torsion. (a 1? a 2 , a 3 , cp 12 , (p 3 ) are the joint variables. 


input activation output 




-.<D— y, 

y 2 

<£)— 


Fig. 2 (a) A single neuron acting as a perceptron. (b) The MLP 
architecture with k inputs and m outputs. 

Concentric Tube Robot Parameters 


Parameter 

Value 

Base Loot ion |uun| 

[0,0,0]' 



Entry vector 

[o, 0 , 1] T 



Section stiffness ratio 

D - 10 



Section 

1/K-i | ltlHl| 

|mm| 

L* [nun] 

proximal (two tubes.) 

242 

150 

150 

distal (one tube) 

194 

100 

250 


The robot kinematics are solved using discretised 
versions of the Equations in [1], for arclength 
discretization of 1mm. 

Forward Kinematics Network 

Training set: By generating 1000000 robot shapes by 
uniformly sampling robot joint variables the set of tip 
poses of the robot and its workspace is covered. 
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Workspace calculation creates the tuples that form the 
training set of the NN. The 5x1 vector encoding the 
position and pointing orientation is used as the target, 
and the joint variables as input. To alleviate bias 
towards small total extensions and degenerate tip 
orientations, small tube translations, i.e. cp < 0.30, are 
ignored, where O is the maximum tube extension. The 
training set is split into training/testing/validation 
(60/20/20) to avoid overfitting. 

Selection of MLP architecture: Kolmogorov’s 
theorem states that any non-linear function can be 
approximated arbitrarily well using an MLP with a 
single hidden layer [5]. Hence, a single-layer MLP (N = 
1) for the forward kinematics is sufficient. 

Cardinality of training set: MLPs were trained for IS I 
= 10 x 10 3 • • • 100 xlO 3 , where ISI is the number of 
training samples, with steps of 10 x 10 3 , for N = 1 and n 
= 10 • • • 150, with steps of 10. 

Inverse Kinematics Network 

NN for inverse kinematics mapping has two differences: 

1) The inputs to the MLP are now the three-dimensional 
position and pointing orientation of the tip, whereas the 
outputs are the joint variables; 

2) It has been proved that approximating the inverse of a 
smooth non-linear mapping, e.g. the inverse of forward 
kinematics, require at least 2 hidden layers [6]; 

The increased complexity introduces the requirement of 
configuration-space splitting. To account for the 
behaviour of inverse trigonometric functions, i.e. arccos, 
arcsin, arctan, which will be implicitly modelled by the 
MLP to account for the torsional rotations of the tubes, 
the configuration space for a, i.e. the tube rotations, is 
split into 4 quadrants, [0, te/2), [tt/2, tt), [tt, 3te/2), [3tt/2, 
2k). This, however, results in an uncertainty in the 
selection of the appropriate NN, since the joint variable 
interval is not known a priori. 

This is tackled by using, for the target tip position and 
pointing orientation, all the trained IK NNs. This results 
in multiple joint variable estimates: a single estimate for 
each tube extension cp i? and 4 estimates for each tube 
rotation a i? i.e., 16 joint variable sets for the robot under 
examination. The correct joint- variable tuple is selected 
as the one that returns the least forward kinematics 
error, since the desired tip pose is known. 

RESULTS 

Forward Kinematics Network 

Figure 3 shows the average, and standard deviation of 
the errors for the tip position (estimated vs actual) and 
tip orientation (estimated vs actual, as the magnitude of 
the orientation vector difference). Increasing the number 
of neurons above 120 and the number of samples above 
60 xlO 3 does not yield precision benefits. With these 
values, the mean tip error is 0.2mm, and the mean 
orientation vector magnitude difference is 0.002. 

Inverse Kinematics Network 

Similar simulations hold for the inverse kinematics 
networks: 10 x 10 3 samples, two layers, and 100 


neurons per layer are sufficient. Then, using the 
described architecture for 1000 samples not considered 
for training results in errors in tube extension below 
0.8mm, and errors in tube rotation below 0.1°. Our 
Matlab implementation calculates the inverse 
kinematics at 50Hz. With hardware implementations 
possible, computational requirements can be trivial. 




Fig. 3 Effect of training set cardinality and number of neurons 
on tip (a) position, and (b) orientation error (as vector- 
difference magnitude). Red vertical lines demonstrate the 
standard deviation of the error. 

DISCUSSION 

This paper presented in simulations the use of neural 
networks to model the forward and inverse kinematics 
of concentric tube robots. This experimentally allows 
the determination of the robot’s motion model by 
observing the tip’s behaviour for different actuator 
inputs, akin to system identification. 
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INTRODUCTION 

The transcatheter approach is gaining popularity for 
many cardiac procedures that have previously been 
performed as open-heart surgery. One of the major 
hurdles of transcatheter interventions, however, is 
providing imaging of sufficient quality for procedure 
guidance and monitoring. 

Ultrasound and X-ray fluoroscopy are the most 
common modalities utilized for the image guidance. 
While both are nonin vasive, each has limitations in 
guiding interactions between the catheter and the tissue. 
Ultrasound images are of limited resolution and suffer 
from high noise. Fluoroscopy provides higher 
resolution, but the images are 2D and contain the 
catheter, but not the tissue. In short, both modalities lack 
ability to provide high-resolution images at the 
interventional site. 

Optical imaging at the tip of the catheter can augment 
the image guidance of these modalities by providing 
intuitive local images of the intervention site. The 
challenge of optical imaging inside the heart, however, 
is seeing through the blood. Three approaches have 
been attempted to overcome this challenge: infrared 
imaging [1], flushing with clear liquid [2] and 
introducing an optical window [3]. In the latter, a 
transparent optical window is pressed against the tissue 
to displace the blood so that the tissue becomes visible. 
The advantages of this approach are that it does not 
require continuous fluid injection into the bloodstream 
and also stabilizes the imaging location through the 
contact force for subsequent tool delivery. 

This abstract describes two of our cardioscope designs 
for intracardiac imaging. The first is a handheld 
instrument for tissue removal. The second smaller 
design, mounted on a robotic catheter, is being 
developed for transapical aortic paravalvular leak 
closure. 

MATERIALS AND METHODS 

Figure 1 shows the two cardioscopes. Their diameters 
are 13mm and 6mm, respectively. The main component 
of both cardioscopes is the transparent optical window 
cast from RTV-2 silicone rubber (QSil 216, Quantum 
Silicones LLC). The optical window encloses the 
camera and LED light sources and incorporates a tool 
channel. The handheld design has an additional channel 
for flushing saline to clear the field of view. 

This work was supported by the the NIH under grant R01HL1 24020. 


Imaging is performed using a 5mm CCD camera 
(Vividia Mini, Vividia Tech.) and 1mm CMOS image 
sensor (NanEye, Awiaba), respectively. The tool 
channels of the two systems were designed for passage 
of cpl.8 mm MEMS tissue removal tool [4] and 4Fr 
closure devices for sealing paravalvular leaks, 
respectively. The field of view of both cardioscopes is 
designed to enable visualization of devices extending 
from the tool channel. 



Fig 1. Cardioscope designs, a) Handheld tissue removal 
instrument, b) Robotic catheter for paravalvular leak closure. 


RESULTS 

The handheld cardioscope was tested in ex-vivo and in- 
vivo experiments. The goal of the both experiments was 
to remove tissue from the infundibulum region of the 
right ventricle, simulating the correction of pediatric 
outflow tract obstruction. 

The ex-vivo experiments were performed on porcine 
hearts connected to a pulsatile pumping system to 
simulate the cardiac cycle. Prior to the experiments, a 
15mm circle was marked on the tissue to specify the 
target for tissue removal, as shown in Figure 2a. The 
cardioscope was inserted through the free wall of the 
right ventricle and tissue was precisely removed from 
the target region as shown in Figure 2b. 
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Fig 2. Ex-vivo experiment, a) Target region of interest marked 
on the inner surface of the heart, b) Target region after the 
experiment, showing precise removal of the tissue. 


For the in-vivo experiment, a 65kg swine was 
anesthetized and the cardioscope was inserted through 
the right ventricular free wall. Figure 3 shows images of 
the ventricular wall during tissue removal. Clear 
visualization of the tissue was established as soon as the 
optical window contacted the tissue. Tissue was 
removed under visual guidance as shown in Figure 3c, d. 



Fig 3. In-vivo View, a) Prior to tissue contact, b) At tissue 
contact, c) and d) Removal of the endocardium. 


The robotic-catheter-mounted cardioscope has been 
tested in bench-top and ex-vivo experiments as shown in 
Figure 4. Figure 4a, b demonstrate navigation of the 
catheter tip inside a beaker of water to “read” lettering 
written on its exterior. In initial ex-vivo experiments, the 
catheter has been navigated inside a water-filled porcine 
heart (Figure 4c). As an on-going imaging 
demonstration project, current experiments involve 
constructing transcatheter valve models incorporating 
paravalvular leaks. The robotic catheter will be 
introduced transapically and the effectiveness of 
cardioscopically-guided navigation to each leak will be 
investigated. 

DISCUSSION 

This paper presents two cardioscope designs with the 
goal of providing tool-tip views inside the heart. The 
designs, which integrate camera, light source and tool 
channel as a single molded unit, are both simple and 
inexpensive to fabricate. As a milestone toward 


percutaneous robotic interventions, a handheld 
instrument was developed and demonstrated in the 
context of beating-heart tissue removal. It successfully 
addressed two key problems: visualization of tissue 
removal and precise control of tool-tissue contact. 

As described, its miniaturization for deployment on a 
robotic catheter has been demonstrated. In combination 
with robotic steerability, cardioscopic navigation is 
currently being explored for paravalvular leak closure. 

It is anticipated that cardioscopic imaging can be 
combined with X-ray fluoroscopy and ultrasound, the 
traditional imaging modalities of intracardiac 
procedures. The ‘local ’ optical image of the intervention 
area from the cardioscope can augment the more ‘global’ 
picture of catheter position from fluoroscopy and 
ultrasound. Together, such imaging may provide 
sufficient additional information to broaden the range of 
transcatheter interventions that can be performed. 



Fig 4. Initial testing of catheter-mounted cardioscope, a) 
Navigating to text on beaker, and b) Cardioscope image, c) 
Ex-vivo navigation into porcine heart, d) Image inside heart. 
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INTRODUCTION 

Evidence has demonstrated manifold differences in 
technical and non-technical skill requirements between 
surgical modalities. With increasing use of Robot 
Assisted Radical Prostatectomy (RARP) for prostate 
cancer it is important that training and assessment be 
evaluated to ensure that they develop surgical skills 
required for safe, competent practice. 

Surgical Safety 

Adverse events arise from technical and non-technical 
skill deficiencies which training and assessment ought 
to rectify. Trainees must develop skills and knowledge 
of generic and procedure -specific techniques such as 
RARP. Adjusting training and assessment to ensure 
these outcomes ensures patient safety and maintains 
competent use of innovative surgical methods. 

Checklists 

Checklist can protect patient safety and measure 
surgical competence. Objective Structured Assessment 
of Technical Skills (OS ATS) has been employed in 
training (gynaecology, ophthalmology), assessment and 
measuring procedural learning curves. To date non- 
specific measures have been designed. One example is 
the WHO Surgical Safety Checklist, acknowledging that 
any operation entails avoidable error. This is employed 
in a multitude of surgical settings (1). Other checklists 
for use in robotic surgery have been designed but do not 
assess specific procedures (2-4). The subjective and 
generic nature of existing assessments indicates the 
need for more specific, objective tools to ensure surgical 
competence in robotic techniques. 

To address this paucity, this study sought to: 

1. Develop and validate a checklist-based training and 
assessment score for RARP 

2. Evaluate learning curve (LC) of trainees using the 
RARP score. 

MATERIALS AND METHODS 
Development & Validation 

This multi-institutional, observational, prospective 
study used HFMEA to identify critical steps associated 
with RARP. HFMEA has been used in high-risk 
industries and employs pre-emptive risk assessment to 


minimize adverse events (Figure 1). Comprising five 
steps, HFMEA combines expert experience with 
brainstorming techniques to list failures and failure 
modes. 

After building a process-map, failure modes and failure 
mode effects for each sub-process are brainstormed, 
constructing an HFMEA hazard analysis table. This 
underwent content validation by experts at a multi - 
institutional meeting in Buffalo, NY. Hazard analysis 
was conducted in an international, multi -institutional 
teleconference with experts to allocate hazard scores 
according to pre-defined definitions (5). 

Content validation was undertaken at several stages 
throughout the process and by circulating the RARP 
Score for use by 17 mentors and trainees, construct 
validity, acceptability, feasibility and educational 
impact were appraised. Construct validity and inter-rater 
reliability was evaluated by two experts using trainees’ 
footage and a video of RARP by an expert surgeon. 

Learning Curves 

15 surgical trainees were scored using the RARP Score 
while operating. Results were analyzed relative to 
RARP experience, examining sub-process learning 
curves. Information regarding the number of procedures 
required to attain competence in a step was uncovered. 

RESULTS 

Development & Validation 

5 surgeons were observed for 42 console hours to map 
key steps of RARP (Fig. 2). 

3 of RARP were identified and consisted of 23 
processes, 64 sub-processes: 

1. Preparation of the Operative Field (9 processes, 12 
sub-processes, 19 failure modes) 

2. Dissection of the Bladder and Prostate (9 processes, 
29 sub-processes, 37 failure modes) 

3. Anastomosis and Closure (5 processes, 23 
associated sub-processes, 27 failure modes) 

HFMEA identified 46 potential causes of “Hazard 
score” >8 and incorporated them into the RARP Score. 
Content validation by multi-national experts (US, UK, 
Europe) developed the RARP Score comprising 17 
stages, 41 steps (Fig. 3). All participants deemed it to 
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encompass important procedural steps. 

Construct validity showed a trend towards significance 
in steps 14, 15, 19, 20 and 28 (P=0.063, 0.093, 0.068, 
0.073, 0.091 respectively). Inter-rater reliability 

demonstrated consistency in marking steps 5, and 27 
(p<0.001). The RARP Score was acceptable, feasible 
and cost effective with educational impact. 

Learning Curves 

15 trainees were assessed for 8 months in 426 cases 
(Range 4-79) with all steps attempted. Most cases were 
T stage 2 (40.3%), N stage 0 (59.9%) and 
“Intermediate” D’Amico risk (36.1%). 

Easier steps (for example patient preparation) are 
undertaken earlier during training and surgeons 
achieve “competence” within few procedures. Learning 
curves demonstrated several findings. There were 
plateaus for Anterior Bladder Neck Transection (16 
cases), Posterior Bladder Neck Transection (18 cases), 
Posterior Dissection (9 cases), Dissection of Prostatic 
Pedicle and Seminal Vesicles (15 cases) and 
Anastomosis (17 cases). For other steps the LC did not 
plateau during data collection (e.g. Expose Prostatic 
Apex and Endopelvic Fascia; 31 cases, Stitching and 
Division of Dorsal Venous Plexus; 32 cases). 


DISCUSSION 

This multi-institutional, international study 
methodically designed the RARP Score using a 
validated risk assessment modality, HFMEA. Content 
validation formed an acceptable, feasible, cost-effective 
and educationally beneficial 17-stage assessment of 
technical competence for hazardous sub-steps of RARP. 
Analyses demonstrated varied LCs. Results can guide 
when surgeons should undertake steps, protecting 
against never events, promoting safety. 


Step 1: 

Define the HFMEATopic 


Trans-abdominal Robot Assisted Radical Prostatectomy 
(RARP) 



Step 3: 

Graphically Describe the 
Process (Mapping) 



MDT meeting 
Anaesthetists, anaesthetic assistant, scrub nurse, 
theatre staff 

7 Surgeons invited for involvement in content 
validation 


Flowchart constructed, illustrating RARP after 42 
hrs observing 5 surgeons and interviewing MDT 
Incorporation of Pasadena Consensus Panel 
recommendations 

Flowchart circulated and revised for international 
content validation 



Identify Failure Modes and Effects and circulate 
hazard analysis table for content validation 
Multi-institutional hazard analysis including 
severity, probability and hazard scores, single 
point weaknessess, detectable failure modes and 
existing control measures 



Step 5: 

Actions and Outcome 
Measures 


Hazard score >8 identified as "key" step 
Detectable steps and existing control measures 
excluded 

Checklist circulated for content validation 


Fig. 1 . HFMEA as applied to RARP. 




Fig. 3. RARP Score. 
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INTRODUCTION 

Breast cancer is the most frequent surgically treated 
cancer in women. The recommended treatment for early 
stage breast cancer is breast-conserving surgery, which 
requires complete surgical excision of the tumor while 
conserving healthy breast tissue. If cancer is found at 
the excision margins (called positive margin), additional 
surgery is performed. Recent reported positive margins 
rates have been as high as 47% [1]. 

At the Hamlyn Symposium in 2014, we introduced the 
concept of a breast-conserving surgery navigation 
system using real time electromagnetic (EM) tracking 
[2]. The system features an EM-tracked tissue locking 
needle to serve as a local coordinate reference (Figure 
1). Wire hooks lock the needle in the tumor, thereby 
achieving accurate tracking when the targeted tumor 
moves and deforms during surgery. The target is 
contoured in EM-tracked ultrasound and defined in the 
coordinate frame of the needle. The tracked surgical 
cutting tool (cautery device) is shown on the navigation 
display relative to the tumor margins. Phantom studies 
with this system have shown 50% reduction in positive 
tumor margins compared to conventional wire 
localization, while total excised tissue amount was equal 
in both groups [2] . In this paper, we present the clinical 
translation to breast-conserving surgery navigation and 
the outcomes of a clinical safety and feasibility study. 

MATERIALS AND METHODS 

Sterility is achieved by placing the EM field generator 
under the sterile drape (Figure 4) and by placing the EM 
sensors in sterile bags. Disposable 3D-printed fixtures 
house the EM sensors. The fixtures are clipped onto the 
needle and cautery (Figure 2). The needle tracking 
fixture is secured to the needle shaft and is compatible 
with a variety of needle brands. We use Sonic Touch 
GPS ultrasound scanner (Ultrasonix, Richmond, BC, 
Canada) with an L14-5gps probe that also includes an 
integrated EM sensor. The mechanical designs, editable 
printable models with user instructions are available on 
the PLUS software website ( www.plustoolkit.org ) [3]. 

For intra-operative segmentation of the tumor volume in 
tracked ultrasound, we model the tumors by an arbitrary 
number of margin points. A convex surface enclosing 
all tumor points is generated by Delaunay triangulation. 
Points are collected during systematic scanning of the 
tumor from multiple directions until the tumor in the 
ultrasound images is encompassed within the margins 
from all directions. Points can be added to update the 


tumor model at any time during the surgery. The cautery 
or any tracked pointer can be used to add points to the 
margin (Figure 3). This feature will be integrated with 
advanced intraoperative tumor detection methods such 
as mass spectrometry and optical coherence tomography 
in the future for tumors not completely visible in 
ultrasound. 

The attention of the surgeons is usually focused on the 
surgical site, so the navigation display is out of their 
direct line of sight (Figure 4). The navigation system 
obtains the surgeon’s attention when the tumor margin 
is breached. We developed a general purpose software 
module to provide visual and/or audio warning signals 
when the tools interfere with predefined tissue areas. 

All software used in our navigation system is part of the 



Fig. 1. Schematic view of the breast-conserving surgery 
navigation system. Blue coils represent EM tracking sensors. 
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Fig. 2. Disposable tracking fixtures designed and 3D-printed 
for the cautery (A, C) and the needle (B, D). Arrows point at 
slots for EM tracking sensors. 
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SlicerIGT open-source framework. The source code is 
freely available from www. slicerigt.org , without any 
restriction of use. All modules are built on the 3D Slicer 
open source application framework ( www.slicer.org) 
and they can be conveniently installed through the 3D 
Slicer extension manager. We provide online tutorials 
for the software modules on the SlicerIGT website. The 
tutorials do not require software coding knowledge. 
New features may be added by minimal Python 
scripting. Devices can be swapped out without 
programming by using PLUS [3]. The resulting 
navigation system was tested in a clinical safety and 
feasibility study on patients with palpable breast tumors. 
The study was approved by our institutional research 
ethics board, and written informed consent was obtained 
from all participating patients before surgery. We 
measured system setup time, calibration time, and the 
total procedure time. Surgeons were surveyed on the 
usability of the navigation system after the operations. 


Questionnaires showed that the surgeons did not find 
the EM sensors to interfere with the procedure and the 
navigation system was generally easy to use. All 
surgeons rated the navigation system as "easy" or "very 
easy" to use, and considered it useful in maintaining 
safe and accurate tumor margins. The system setup and 
calibration time decreased from 15 minutes in the first 
case to only 6 minutes in the last case. Procedure times 
fell within the normal breast-conserving surgery times 
at our institution (Table 1). 


Number of patients in study (N) 

6 

Age (range) 

29 - 92 yrs. 

Anesthesia (sedation / general) 

3/3 

Sentinel node biopsy in procedure (N) 

4 

Operation time (avg. ±SD) 

51 ±16 min 

Navigation setup time (avg. ±SD) 

8.7 ±1.4 min 

Margin histology (negative/positive) 

6/0 


RESULTS 

Six patients were included in the safety and feasibility 
study with stage I A to III A breast cancer. All margins 
were histologically confirmed as negative. There was no 
complication due to the navigation system, and no 
breach of sterility was detected in any of the cases. 
Clinical features and results of the study are shown in 
Table 1. 



Fig. 3. Extending the tumor margin during surgery. The 
surgeon points at the tumor location with the cautery tip (left). 
The navigation software adds a new tumor point and extends 
the tumor margin with the new point set (right). 


» ^ 



Fig. 4. The navigation system components (ultrasound and 
position tracker) are under the sterile draping. The navigation 
display is placed in front of the surgeons during the operation. 


Table 1. Clinical features and results of the clinical safety and 
feasibility study. 

DISCUSSION 

Our results suggest that real time navigation with EM 
tracking is safe and feasible in breast conserving 
surgery. No navigation system specific technical 
complications were experienced. 

One person was needed in addition to the standard 
surgical staff to operate the navigation software. The 
surgeons indicated a strong need for sterile human- 
computer interface. We are planning to solve this 
problem by adding wireless touch-screen tablet in a 
sterile bag. With this, several workflow steps can be 
performed by the surgeons using the tablet device, such 
as tumor contouring and virtual viewpoint adjustment 
for the 3D navigation display. In one occasion the EM 
tracker slipped off of the needle and needed re- 
clamping, but tracking remained accurate because the 
fixture is clamped on the needle next to the hub 
reproducibly. 

The breast surgery navigation system performed without 
any technical and clinical failure. A larger clinical study 
on non-palpable tumor cases is currently underway. 
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INTRODUCTION 

The fusion of Magnetic Resonance Imaging (MRI) and 
Ultrasound (US) for targeted prostate biopsy can solve 
the diagnostic dilemma of patients with repeated 
negative prostate biopsies seen in the conventional 
Trans-Rectal UltraSound (TRUS) guided systematic 
biopsy. Recently, 68 Gallium labeled ligand of Prostate 
Specific Membrane Antigen ( 68 Ga-PSMA) Positron 
Emission Tomography (PET) was introduced that, in 
conjunction with MRI, provides combined molecular 
and structural information for the detection of Prostate 
Cancer (PCa) [1]. Hence, we developed an open source 
framework [2] that combines the preoperative PET/MRI 
images with TRUS and provides multimodal image 
guidance for targeted biopsy. In this paper, we present 
the technical challenges in the development of 
multimodal image guided prostate biopsy, especially in 
3D TRUS acquisition and multimodal image 
registration. Further, we explain the steps to address 
these specific challenges and some unsolved problems. 
Finally, we discuss the clinical evaluation of the system. 


MATERIALS AND METHODS 

The multiparametric MRI and 68 Ga-PSMA PET were 
acquired on a combined PET/MR system (Siemens 
mMR) with 3T magnetic field strength. The 3D TRUS 
acquisition and biopsy were carried out in the urology 
department of our university hospital. We used a US 
system (Hitachi AVIUS) with a 2D front fire trans- 
rectal probe. However, our system is independent of the 
US machine as we are using a video frame grabber 
(StarTech) for image acquisition in HD (1280 x 1024 
pixel). The US probe is tracked by an optical tracking 
system (NDI Polaris). We use a workstation with 2 Intel 
Xeon processors running at 2.13 GHz with 32GB RAM 
and a graphics card (Nvidia GeForce GTX Black Titan). 

The clinical workflow has three steps - (i) pre- 
interventional 68 Ga-PSMA PET-MRI image acquisition, 
(ii) TRUS acquisition, 3D reconstruction and image 
registration, and (iii) biopsy as shown in Fig. 1. The 
acquisition of 68 Ga-PSMA PET-MRI for prostate cancer 
imaging is explained in [1]. 



Step2: 

Multimodal 

Image 

Registration 



US 

in green 


Step3: 
Multimodal 
Image-guided 
prostate biopsy 



Needle 
guidance 
PET- US 


Fig. 1 Steps in clinical workflow. 

3D TRUS acquisition 

The challenges in 3D TRUS acquisition are precise 
tracking of the 2D US probe, accurate spatial calibration 
of the US probe for different depth settings, and reliable 
3D compounding in the standard anatomical axes. 

The spatial calibration of the ultrasound probe, 
acquisition of tracked ultrasound, and 3D compounding 
are done by the fCal application available in PLUS 
framework [3]. We use a reference target attached to the 
biopsy chair, as described in [2], that facilitates the free 
movement of tracking camera and reorientation of the 
3D TRUS volume along the standard anatomical axes. 

Multimodal image registration 

The accurate modelling of the prostate deformation 
within clinically acceptable time limits is the main 
challenge for image registration and successfully 
targeting suspicious lesions. The review of various 
commercially available fusion devices is available in 
[4]. Most of these commercial devices use rigid image 
registration. The state of the art research on TRUS/MRI 
image registration is summarized in [5], wherein the 
typical approaches are either surface- or model-based. 
We use surface based deformable registration using 
Coherent Point Drift (CPD) algorithm [6] and compare 
it with rigid anatomical landmark based registration. 
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RESULTS 

Phantom studies 

For the evaluation of surface based elastic registration, 
we acquired MRI and TRUS images of a multimodality 
prostate phantom (model 053-MM, CIRS) that has 
visible structures such as urethra (7 mm) and three 
randomly placed lesions of approximately 5-10 mm. Six 
landmarks were annotated by an expert on images from 
both modalities. 

The mean landmark registration errors (LRE) for 
surface based CPD method and rigid registration were 
1.14 mm and 1.87 mm, respectively. The comparison of 
registration by both methods in the form of axial and 
coronal views is shown in Fig. 2. 


Rigid Surface-based (CPD) 



Fig. 2 Surface based elastic algorithm shows better alignment 
compared to rigid registration in phantom studies. 

Retrospective study on patient dataset 


Rigid Surface-based (CPD) 



Fig. 3 Surface based elastic algorithm shows better alignment 
compared to rigid registration in patient data. 

The 3D TRUS and 68 Ga-PSMA PET-MRI images were 
acquired as per the method described earlier. The expert 
performed manual segmentation on MRI and TRUS 
images for surface extraction and annotated four pairs of 
anatomical landmarks in each case for the evaluation of 
registration. 

We evaluated our registration method on five patient 
datasets. The mean LRE for surface based elastic 
registration was 2.49 mm in comparison to 4.63 mm for 
rigid registration. The outcome of both registration 
approaches for two patient cases is shown in Fig. 3. 


Exemplary clinical case 

After validation of the system, we used it in one 
multimodal image-guided biopsy. A 65 year-old patient 
with previous negative biopsy and rising PSA was 
referred for 68 Ga-PSMA PET-MRI examination and 
follow-up biopsy. In addition to the 10-core systematic 
biopsy, two targeted biopsy were taken from a 
suspicious region in the left apical zone under 
multimodal image guidance. 

The histology confirmed significant PCa with a Gleason 
score of 3 + 4 = 7. The targeted biopsy cores showed the 
highest percentage (90%) in pathological tissue. 

DISCUSSION 

The multimodal image guidance shows a value in the 
detection of significant PCa, especially in men with 
previous negative biopsy and rising PSA. Though we 
tried to address some of the challenges in 3D TRUS 
acquisition and multimodal image registration that can 
be done online within short time ~5 min, some other 
issues in the tracking and image registration still need 
further work. The tracking corruption can be improved 
by the preprocessing of tracking stream with kalman 
based estimation. The image based deformation models 
can improve the registration in comparison with surface 
based methods. 
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BACKGROUND 

An optimal view on the operating field is important for 
high quality performance in laparoscopic surgery. 
Nevertheless, due to limited personal resources, 
inexperienced surgical staff members often perform the 
camera guidance. Therefore, research into camera 
robots began over 15 years ago [1]. However, until 
recently these camera robots were not used in clinical 
routine due to cumbersome human-machine-interfaces 
[2]. Up to now camera robots lack real cognitive 
capabilities. To address this need, we developed a 
robotic camera assistant that learns how to guide the 
camera based on the surgeon’s preferences and acts as a 
cognitive surgical assistant (CoS A). 

METHODS 

With the OpenHELP phantom n=20 laparoscopic rectal 
resection with total mesorectal excisions were 
performed [3]. The laparoscopic video as well as the 3D 
position of the surgical instruments and the laparoscope, 
obtained with the Polaris® optical tracking system (NDI 
Inc., Waterloo, Canada), were recorded. Data annotation 
was performed with the s.w.an-Suite software (SWAN 
GmbH, Leipzig, Germany). Surgical experts classified 
the camera guidance quality of the human camera 
assistant as good, neutral or poor by rating each video 
frame. Furthermore, all parts of the surgical procedure 
were classified according to a surgical process model 
[4] comprising 5 phases and 15 steps. These semantic 
annotations as well as the 3D coordinate data were 
linked to each video frame to populate the knowledge 
base. Synthetic data was computed to provide additional 
learning examples of poor endoscope positioning for 
machine learning. This knowledge base was further 
expanded by using supervised machine learning 
(Random Forests) to train camera quality classifiers 
specific for each step [5]. 

The camera position was constrained by universal 
surgical requirements such as remote center of motion 
(respecting the trocar) and smoothness of movement. 
However, the actual robot movement was controlled 
solely through the learned camera quality classifiers. No 
additional movement rules were programmed. Thus, the 
movement was solely based on the instrument positions, 
providing flexibility during each step. Also, the 
different steps of the surgical procedure could be 


performed in flexible order. The software was 
developed within the ROS-framework [6]. 

Exemplarily two surgeons performed the mobilization 
of the descending hemicolon as a step in rectal resection 
using the OpenHELP phantom. As robotic platform two 
different systems were used, ViKY® (EndoControl 
S.A.S., Grenoble, France) and LWR (KUKA AG, 
Augsburg, Germany). 

RESULTS 

Altogether 151,598 video frames and 552,711 3D 
positions were recorded and semantically annotated. 
Additionally 154,018,480 synthetic 3D positions were 
computed. Based on the results of the camera quality 
classifiers (figure 1), the robot moved the camera to the 
optimal position in real time. Surgery could be 
performed by two different surgeons on both robotic 
platforms (figure 2). The surgeons were able to perform 
the surgery without any manual corrections in both 
cases. 

CONCLUSIONS 

For the first time our knowledge-based approach 
brought cognitive capabilities to laparoscopic camera 
robots. Both camera robots assisted the surgeons 
autonomously, their movement was smooth and the 
impression of the camera quality was good. A thorough 
evaluation of the camera guidance based on various 
subjective and objective criteria is underway. 

The software worked independently of the robotic 
actuator and will be integrated in the framework of the 
CoSA envisioned in the Collaborative Transregional 
Research Center Cognition-Guided Surgery. 
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Fig. 1 Visualization of the camera quality classifiers results. 
Green, yellow and red arrows indicate good, neutral and poor 
camera positions, respectively. 



Fig. 2 Surgeon performing a laparoscopic procedure with a 
cognitive camera robot based on the KUKA LWR-platform. 
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INTRODUCTION 

In state-of-the-art Trans-oral Laser Microsurgeries 
(TLM), laser aiming and surgical outcome rely on the 
skill and dexterity of the surgeon. A manual 
micromanipulator, consisting of a beam-splitter mirror 
and a small joystick, attached to the surgical 
microscope, aims the laser beam at the surgical area (the 
vocal fold, Fig. 1 (left)). Accurate laser aiming is critical 
in preserving healthy tissue while also removing the 
pathology. In recent years, motorized laser scanners 
have offered greater aiming accuracy and efficiency 
using automated incision patterns [1]. 

This paper proposes a novel design for motorized laser 
micromanipulation in TLM, based on a 2 degrees-of- 
freedom (DOFs) spherical orienting device for the 
beam-splitter mirror itself. The combination of this 
device with an integrated, computer-assisted surgical 
system provides precise aiming control and automated 
intraoperative planning, with dynamic active 
constraints, making it suitable to clinical applications. 


Fig. 1 Left: State-of-the-art TLM setup. 

Right: The new micromanipulator - X and Y axes. 

MATERIALS AND METHODS 

The state-of-the-art TLM has a beam-splitter mirror 
aligning the beam from the laser focusing optics to the 
microscope’s optical path and reflecting it towards the 
surgical site (Fig. 1 (left)). The motorized 
micromanipulator therefore had to allow: (i) 
compatibility with the laser focusing optics and 
coupling with the surgical microscope; and (ii) 
maintaining the surgical line-of-sight through the optical 
path of the microscope and the mirror. Therefore, the 
new design was based on actuating the beam-splitter 
mirror itself (Fig. 2). 


New Micromanipulator Design 

Two voice-coil linear actuators (SMAC CAL 12, 24V, 
1.5 A) drive a 2-DOF spherical orienting mechanism, 
where the linear motions are converted into spherical 
orientations of the mirror (roll + pitch) resulting in 2- 
DOF motion of the laser at the surgical site (Fig. 2). 

1. The design includes a hollow core supporting 
(HC) structure, having two re volute support joints. 
Their respective axes of rotation pass through the 
spherical center of rotation of the structure in 
mutually orthogonal disposition. Two actuation 
levers are attached to the structure through the 
re volute joints. 

2. The back-n-forth motion of each lever generates a 
roll and pitch of a mirror-holding (MH) structure 
about the respective rotation axes. 

3. For the 4 X’ axis, the mirror rolls about the central 
axis of MH, while for the ‘Y’ axis, the mirror 
pitches about the tangential axis of MH. 



Fig. 2 The new motorized micromanipulator design rendering. 

The light-weight linear actuators offer a 10 mm stroke 
length and a 1 pm encoder resolution. The actuators 
were sized to provide fast motion of the mechanism 
without high inertia. The actuators are controlled 
through a custom-designed motor control board which 
includes a velocity-based trajectory-control method for 
the programmable laser trajectories. 

Integrated System 

The prototype was fabricated and integrated into the 
surgical system, introduced in [2]. It consists of the 
following components and features: 

1. Motorized laser manipulator: As described above. 

2. Input: A graphics stylus with tablet interface for 
intuitive laser aiming and activation. 

3. Visualization: Two HD cameras (Prosilica 
GT1910) attached to the surgical microscope 
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acquire stereoscopic video. They are displayed in a 
modified head-mount display, the Virtual 
Microscope , providing a comfortable and 
ergonomic visualization of the surgical site. 

4. Configuration interface: A touchscreen interface to 
define types of surgical actions, view messages, 
and capture important procedural information. 

5. Assistive features: 

a. Free-hand aiming and cutting with the laser 

b. Intraoperative planning: Virtual lines and 
shapes defined for automatic incisions. 

c. Virtual constraints: Virtual go / no-go zones 
defined to deactivate the laser for safety. 

d. Assistive techniques including optical 
magnification, and gesture / motion scaling. 

System calibration for this new mechanism with the 

surgical system was adopted from [1]. 


RESULTS 
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Fig. 3 The experimental setup for user and surgeon trials. 


Characterization trials of the new system resulted in the 
values listed in Table I. The resolution offered by the 
new system is less than half the smallest laser spot size 
obtained from the focusing optics (110 pm), allowing 
for precise laser incisions. 


Table I Characterization of the new system 


Operating Distance 

400 mm 

Surgical Range 

8640 sq. mm (64 x 135) 

Maximum Speed 

7.09 m/s 

Resolution 

12.8 pm (X-axis); 

27.0 pm (Y-axis) 


User accuracy trials with 10 non-medical subjects were 
conducted, where each subject traced 6 different 
randomized shapes (straight lines, S-curves, C-curves), 
comparing both systems - the traditional manual system 
and the new design (Fig. 3). The system accuracy was 
assessed using root-mean-square-error (RMSE) and 
Maximum Error metrics. The execution times metric 
provided a measure of the mental workload (Table II). 
The new design produces a statistically and significantly 
more accurate performance than the traditional manual 
system, showing a more than 57% improvement. The 


new design, in combination with the computer-assisted 
system, offers superior performance. 


Table II Comparison of Accuracy Metrics 



Traditional 

System 

New System 

RMSE (mm) 

0.3508 ±0.1756 

0.1499 ± 0.0665 

Max. Error (mm) 

1.5358 ±0.8501 

0.4444 ± 0.1798 

Times (sec) 

14.4031 ±7.5778 

10.7226 ± 2.3306 


These results indicate that the new system reduces 
complexity from the user’s perspective and its uniform 
and consistent integration into the surgical procedure 
could lead to its successful clinical acceptance. 

Ex-vivo trials were conducted with 3 expert surgeons 
performing near-real surgical tasks (incisions and 
ablations) on pig larynxes. The surgeons were also 
encouraged to use the regular surgical instruments, e.g., 
forceps that are part of the surgery in the operating room 
(OR). The following qualitative feedback was received 
from the surgeons: 

1 . The mechanism allows a larger operating distance 
and surgical range, providing easier passage and 
maneuvering of surgical instruments. 

2. The smooth control and accurate motion of the 
laser using the graphics tablet improves the cut 
quality and reduces carbonization. It is intuitive 
and easy to use. 

3. The intraoperative planning mode is useful for 
quick and precise cuts. The ability to perform long 
incisions and spot ablations automatically is very 
useful. 

DISCUSSION 

An earlier prototype by the authors was based on a fast 
steering mirror (FSM) which allowed faster scans (10 
m/s) and greater resolution (4 pm), but had to be placed 
between the focusing optics and the beam-splitter mirror 
[1]. Additionally, the small diameter of the FSM, the 
allowable focal distance, and the type of focusing optics 
were important limitations in that design. These served 
as motivation for this new design based on motorizing 
the beam splitter mirror itself. The spherical orienting 
mechanism overcomes the earlier limitations, and offers 
a maximum speed of 7 m/s and a functional resolution 
between 12.8 and 27 pm, comparable to the FSM 
system [1]. It achieves greater than 57% improvement 
in accuracy indicating that the mechanism can greatly 
improve surgical safety, and comparing favorably to the 
FSM prototype (around 69%, [1]). The new system was 
evaluated positively in ex-vivo trials with expert 
surgeons pointing to its suitability in the OR. 
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INTRODUCTION 

Innovation in surgical robotics has seen a shift away 
from traditional rigid systems (such as Intuitive 
Surgical’s daVinci™ platform) towards flexible systems 
that can access remote locations inside the body. 
However, the flexible nature of these delivery systems, 
coupled with a lack of haptic feedback, could result in 
the application of unregulated forces that could damage 
or perforate tissue in delicate intraluminal spaces. 

Several groups have recognized the need for distal 
force sensing in flexible surgical tools and robotics [1, 
2]. However, a general reliance on conventional 
fabrication techniques ultimately limits the complexity 
and sophistication of distal implementations of such 
sensor systems, and poses a barrier to further innovation 
and widespread adoption. Herein we present a mm-scale 
force sensor manufactured using a composite lamination 
fabrication process, wherein linkages pre-machined in 
the laminate provide the required degrees -of-freedom 
and fold patterns to facilitate self-assembly. Using 
purely 2-dimensional batch fabrication techniques, 
energy contained within a planar elastic biasing element 
directly integrated into the laminate is released post- 
fabrication, allowing the sensor to v self-assemble' into 
its final 3 -dimensional shape. The geometry of the 
sensor was selected based on size constraints inherent in 
minimally-invasive surgery, as well as with specific 
focus on optimizing the sensor's linearity. The sensor is 
unique from fiber-based force sensors in that the emitter 
and the detector are encapsulated within the sensor 
itself, allowing it to be retrofitted onto existing tools. 

MATERIALS AND METHODS 

The sensor operates by the principle of light intensity 
modulation ( LIM ), a transduction method which has 
already been FDA-approved (St. Jude Medical’s 
TactiCath force-sensing ablation catheter). The 
principle works as such: a light-emitting source 
(infrared LED) is separated from a detector (infrared 
phototransistor) by some distance as determined by an 
elastic mechanism. As a force is applied, the elastic 
mechanism deforms, bringing the source closer to the 
detector. By measuring the output current of the 
detector, the input force can be determined. 

Up until now, due to difficulties in manufacturing 
and assembly of the requisite components, this modality 
has been practically realized by transmitting the light 
from a proximal source through optical fibers to the 
distal end of the tool where the force is being applied [3, 



Fig. 1 Image of force sensor molded onto a 6F mock catheter, 
where the detail shows the sensor self-assembling as a result 
of a pre- stressed biasing spring contained within. 

4]. Other light-based approaches (Fabry -Perot 
interferometry, fiber-Bragg gratings) also require optical 
fiber transmission as well as expensive interrogation 
hardware [5, 6]. The drawback of fiber-based 

approaches is that the tool must be designed ‘around’ 
the optical fibers and, as such, is not a modular solution. 

We have designed, fabricated and tested a modular 
LIM force sensor (see Fig. 1) that fully contains the 
emitter and detector and is compliant with the strict size 
requirements of endoscopic working ports (2.8mm OD). 
Using a manufacturing process based on printed circuit 
board fabrication, several layers of structural (304 
Stainless Steel), flexible (Kapton polyimide), adhesive 
(DuPont FR1500 acrylic adhesive), and conductive 
(copper) material were laser-machined and laminated 
together. A pre-stretched spring (a) provides a restoring 
force that causes the sensor to ‘self-assemble’, much 
like a pop-up book, (b) serves as the elastic mechanism 
which deforms upon force application, and (c) serves as 
a grating which allows more light through as the spring 
is expanded to increase sensitivity. Fig. 2 shows an 
overview of the manufacturing process. 

RESULTS 

The sensor was calibrated using a reference load cell to 
generate the calibration curve shown in Fig. 3. The 
sensor exhibits linearity over O-lOOmN range, after 
which, an exponential relationship more accurately 
describes sensor behavior. The calibrated behavior 
closely matches the predicted behavior based on a 
simple point- source emitter model. The sensor exhibits 
sub-mN resolution which was quantified by integrating 
the power spectral density of the sensor output given no 


27 



Fig. 2 Manufacturing process used to fabricate self assembling sensor: (1) multiple individually -machined layers of material are 
laminated to form a composite, (2) release cuts are made using a laser, (3) the alignment scaffold is discarded, (4) the sensor is 
submerged in an ultrasonic bath which causes assembly spring to retract and assemble the sensor, (5) pick-and-place components are 
reflow soldered to complete assembly. A scanning-electron microscopic (SEM) image shows sensor detail. 
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Fig. 3 (left) fabricated sensors, both pre- and post-assembly, 
(right) sensor calibration curve, compared with a linear and 
exponential fit, as well as a comparison with the deterministic 
model based on the optoelectronic properties of the system. 


Table 1 Sensor Specifications 


Parameter 

Value 

Unit 

% FS 

Sensitivity (0-100 mN) 

5.0 

V/N 

N/A 

Range 

0-200 

mN 

N/A 

Resolution 

0.8 

mN RMS 

0.4 

R 2 (Exponential) 

0.99 

N/A 

N/A 

R 2 (Linear, 0-100 mN) 

0.97 

N/A 

N/A 



Fig. 4 In-vitro palpation test, showing a significant 
discrepancy between the haptic force surgeon-side and the 
distal force measured by the sensor. 


can detect sub-mN forces over a range of 0-200 mN. An 
in vitro palpation simulation showed that the sensor has 
the sensitivity necessary to extract ~mN-level distal 
forces from an otherwise contaminated haptic profile, 
indicating the need for distal sensing modalities for 
obtaining accurate information from the tool-tissue 
interface. Future work will focus on optimizing the 
sensor design, improving methods of encapsulation and 
integration, and investigating multi-axis capabilities. 


input stimuli. The sensor’s sensitivity can be controlled 
by selecting the appropriate thickness of the spring 
layer, and also by encapsulating the sensor in various 
polymers or rubbers which are IR-transparent. Sensor 
specifications are given in Table 1. 

The sensor’s efficacy was determined in vitro by 
encapsulating the sensor in silicone rubber, integrating 
the sensor onto the distal end of a 6F catheter and 
performing a palpation task on a block of silicone 
rubber (EcoFlex 00-50, Smooth-On) through a tortuous 
GI tract analog (rubber tubing bent at a 180-degree 
angle). A reference load cell measures the insertion 
force (a combination of the palpation force and 
frictional forces). The results are shown in Fig. 4. The 
distal sensor is necessary to discriminate distal forces at 
the proximal end due to friction in the system and other 
sources of contamination. 

DISCUSSION 

The results presented herein indicate a promising 
solution to developing robust, low-cost, modular, 
millimeter-sized force sensors with the resolution 
necessary for delicate MIS surgical interventions. 
Preliminary calibration tests have shown that the sensor 
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INTRODUCTION 

Colorectal cancer is the third most common cancer for 
both men and women, and second leading cause of 
cancer death [1]. Traditional surgical treatment required 
complete anterior resection, i.e., removal of the lower 
rectum. Advances in endoscopic surgery allow local 
excision of rectal adenomas and early stage carcinomas 
through a transanal approach [2-4] . This is a technically 
demanding procedure since the surgeon must operate 
via the narrow opening of the anus; sometimes to a 
depth of 1 8cm or more. 

The aim of the present work is to lower the technical 
barriers to surgeons’ adoption of transanal excision. 
The solution is to perform the operation telerobotically. 
This delivers better visualization, access, and control of 
instruments to the surgeon. A previous study used a full 
daVinci robot [5], which has poor operating workspace, 
and form factor for transanal operation. This paper 
describes a robot specially designed to target this 
procedure. 

The Micro-IGES (Microscopic Image Guided 
Endoluminal Surgery) robotic surgical system has been 
developed to perform transanal excision under 
teleoperative surgeon control. The high dexterity 
wristed 7-DoF (Degrees -of-Freedom) tools give the 
surgeon robust manipulation capabilities. System 
design is related to other snake-like systems, but to the 
authors’ knowledge this is the first system designed and 
tested for colorectal surgery. 

The remainder of this short paper introduces the Micro- 
IGES system and presents results from pre-clinical in- 
vivo and in-vitro tests. 

SYSTEM DESIGN 

Mechatronics - Seven DoF tools provide Cartesian 
position/orientation and grasping capability. Tools are 
28cm long with 5mm diameter. Insertion along major 
axis of the rectum uses a rack and pinion mechanism. A 
rotational joint rotates the gripper-and-tool shaft. 
Lateral position of the end effector uses two snake-like 
joints with two segments each. A 3 -DoF end effector 
provides two degrees of orientation plus grasping. 
Stainless steel cables transmit mechanical power from 
motor capstans to the tool end effector. Rack and 
pinion system is direct drive and tools ride on a rail 
mount so that they can slide in and out. 


Actuation for the mechanism uses an "actuation pack" 
attached to the external end of the tool. It houses seven 
13mm Maxon brushless motors with 67:1 gear ratios 
and quadrature encoders sensing 250 counts/rev. It uses 
custom designed, small form factor motor controllers 
that communicate with the host PC through a shared 
RS-485 bus. Motor controllers run a PID control loop 
to achieve a target motor angle (encoder count) 
specified by the host. Each pack also houses a cooling 
fan and DC/DC converter. 

Robot Control - Robot is controlled from a standard 
Linux PC with PCIe RS-485 card. Eight mega-baud 
bandwidth provides 180-250 Hz input/output rate with 
fourteen motor controllers. Master- slave control uses 
direct joint mapping for insertion, tool roll, and wrist 
axes. A rotational transform ensures that up/down and 
left/right motions are preserved throughout tool rotation. 
Motion scaling is applied to tool translation. 

Surgical master comprises two Force Dimension 
Omega.7 haptic devices and a footpedal. No force 
feedback is provided to the surgeon. Visual feedback is 
from an Olympus steerable 3D stereo endoscope. 

Surgical Site Access - Transanal surgery requires access 
to and insufflation of the rectum. The Micro-IGES uses 
a port with four insertion points for a camera, two 
robotic tools, and a third robotic or manual tool. The 
robot mounts directly to the port. Both are held by a 
support arm. Pneumorectum is maintained by an airtight 
silicon gasket. The seal was able to maintain pressure 
of 20mm Hg with minimal leakage. 

The supporting arm is a 6-DoF passive actuated serial 
link arm. Each of the six degrees of freedom is 
equipped with a brake and are together capable of 
holding up to 5 Kg. A momentary pushbutton switch 
disables brakes and allows the surgeon to manually 
reposition the arm. Releasing the button engages the 
brakes, freezing the arm's configuration. 



Fig. 1 Micro-IGES system CAD rendering. 
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Fig. 2 Surgical master station. Video shows rectal wall tissue 
under tension. 


Operating Theatre Integration - The entire system was 
designed for use with medical equipment necessary for 
clinical use. A Karl Storz Electronic Insufflator 
connected to the port provides C0 2 insufflation. The 
surgical area can be ventilated through valves attached 
to the port; this is sometimes necessary to clear smoke 
from electrical diathermy operation. An Erbe VIO 
200D powers the diathermy tool. Suction/irrigation can 
be applied through an integrated lumen in the robotic 
diathermy tool or using an additional, manual tool 
inserted through the fourth insertion point in the port. 

Altogether this comprises a nontrivial OR footprint. 
However, the space requirements are still less than other 
commercial surgical robotic systems; much of the 
equipment is already present during conventional non- 
robotic operation; and the system was tested in theatre 
and proved it could be accommodated. 

EVALUATION STUDIES 

Benchtop testing - Initial testing of the system in 
benchtop dexterity tests showed capabilities for 
environment manipulation and suturing. 



Fig. 3 In operating theater ready to operate on draped porcine 
patient. Shown are airtight port, tool mounting rails, actuation 
pack, and endoscope. Insufflation attachment at sides. 



Fig. 4 Full thickness resection of rectal wall showing (left to 
right, top to bottom) preoperative rectum, circumferential 
marking with diathermy, resection under tension (2), and 
completed full thickness resection. 

Suturing was carried out on a tissue phantom suturing 
pad. Using just the robot and endoscopic visual 
feedback two sutures were tied in twenty minutes. 
These were the first and second knots tied by the 
system. 

In-vivo testing - The complete clinical system was 
deployed in animal theatre and a simulated tumor 
excision was performed. The TEMS clinical procedure 
protocol was followed, summarized as: 

• Patient received general anesthesia; 

• Patient positioned and draped; 

• Insertion of port using obturator and positioned 
with positioning arm; 

• Insertion of endoscope and robotic instruments; 

• Circumference to be excised is marked with 
diathermy; 

• Full thickness excision performed using grasper 
and diathermy; 

• Removal of tools and scope; 

The robot under surgeon control proved capable of 
excision of a simulated rectal wall tumor. Closure was 
not carried out. 

DISCUSSION AND FUTURE WORK 

The robot has proven capable of tumor excision and 
suturing. Work still remains to improve control so robot 
motion is more natural and easy for the surgeon. 
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BACKGROUND 

The Fundamentals of Laparoscopic Surgery (FLS) 1 is a 
training programme designed to equip residents with the 
requisite knowledge and technical skills to perform 
Laparoscopic Surgery (LS). Since FLS is a mandatory 
criterion for general surgery board certification in the 
United States, it sets the standard in an era when 
surgeons are increasingly expected to have documented 
evidence of competency. Clearly, if a surgeon fails the 
FLS test is it unlikely that he or she has acquired the 
necessary ‘fundamentals’ for safe LS performance, but 
what about those who pass the test? What does reaching 
expert-derived benchmarks say about one’s level of 
familiarity with the skill or more specifically the degree 
to which the technique has been engrained? The 
prefrontal cortex (PFC) is a brain region known to be 
associated with attention and concentration and an area 
hypothesized to attenuate in-line with increasing 
technical skills acquisition and hence a useful surrogate 
for magnitude of task internalization 2 " 5 . 

METHODS 

45 subjects of varying surgical experience were 
investigated using a 44-channel Optical Topography 
(OT) device (ETG-4000, Hitachi Medical Corp) to 
monitor changes in prefrontal responses during a 
laparoscopic suturing drill. To characterize frontal brain 
behaviour over the time of course from early learning to 
automation both cross-sectional and longitudinal studies 
were conducted as follows: (a) a cross-sectional study 
compared PFC excitation in 35 surgeons (novices with 
2h training: n=12; registrars/residents: post-graduate 
year (PGY) 3-5, n=12; expert laparoscopists: n=ll), and 
(b) a longitudinal study tracked changes in PFC 
responses across four time points as a second cohort of 
novices (n= 13) acquired suturing skills across a week 
of practice (8 hours). Laparoscopic suturing skill was 
objectively evaluated using the FLS framework (time, 
penalties, etc). Cortical haemodynamic data was 
analysed using a region of interest analysis (ROI). 

RESULTS 

Laparoscopic Technical Skills 

Superior technical performance was observed in experts 
compared to residents (median FLS scores ± IQR: 
experts = 487+53, residents = 400+90, Mann-Whitney- 


U p<0.010), who in turn outperformed novices (2 hours 
training=3 16+74, Mann-Whitney-U p<0.001). 
Deliberate practice led to significant improvements in 
technical performance (median FLS scores + IQR: 
initial session (2 hours training = 302+128, mid session 
(5 hours training = 421+86, final session 8 hours 
training =471 + 40, Friedman p<0.001) followed by 
skill stabilization when assessed at retention (1 month 
after cessation of practice = 468+56, Wilcoxon-Sign- 
rank p=0.808). At practice termination, technical skills 
amongst experts and trained novices could not be 
differentiated (p=0.014), based on a statistical threshold 
following Bonferroni correction p<0.008. 

Cortical Brain Function 

An inverse relationship was observed between the grade 
of the surgeon and PFC excitation [AHb0 2 (pM x cm) 
median + IQR: novices = 10.19+21.75, residents = 
7.46+ 12.57, experts = 1.56+11.37, p<0.001]. However, 
despite progression toward near expert levels of 
suturing skill, additional training failed to lead to 
progressive PFC attenuation [AHb0 2 (pM x cm) median 
+ IQR: initial = 8.08+19.95, mid = 7.67+21.12, final = 
7.77+17.91; retention = 8.88+19.92, p=0.065]. As 
illustrated in Figure 1, at practice termination, suturing 
evoked a response that was on average four to five times 
greater among trained novices than expert 
laparoscopists, despite similarities in suturing skill. 



Fig. 1 Graphical illustration depicting the disparity in PFC 
activation associated with expertise (top row) and lack of 
progressive PFC attenuation associated with a week of 
practice (bottom row). Group median value (pM x cm) for 
intensity of activation is depicted on a colorimetric scale. 
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CONCLUSION 

FLS proficiency criteria based on experts’ motor 
behaviour or technical skill such as time and accuracy 
reflect a one-dimensional performance assessment. 
Current data suggests that despite attaining these 
benchmarks, trainees need significantly more attention 
and concentration than experts, and hence may not have 
internalized or automated the task. 
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INTRODUCTION 

Suturing is an essential part of surgical procedures, yet 
tension regulation is difficult when using tele-operator 
systems [1]. Incorrect tension application can lead to 
various complications. It has been suggested that the use 
of haptic feedback to allow tension perception would 
reduce such errors [1]. Many groups have highlighted 
the benefit of haptic feedback in robot assisted 
minimally invasive surgery (RMIS) and bi-lateral, 
kinesthetic ‘force feedback’ solutions have been 
proposed. However, instability issues have prevented 
such solutions from reaching clinical implementation 
[2]. An alternative to bi-lateral force feedback is the use 
of sensory substitution to represent haptic information 
to the surgeon without influencing system stability 
[ 1 ] [2] . Examples of sensory substitution include visual 
and vibrotactile feedback, though it has been suggested 
that such techniques, which do not represent forces by 
natural methods and/or channels, may lead to higher 
cognitive loading [2]. In our work, the use of fingertip 
surface deformation (also known as tactile skin stretch) 
is considered as a method of representing tangential 
(shear) and normal (indenting) interaction forces 
through a natural haptic feedback method. Skin stretch 
feedback avoids the controller instability issues of bi- 
lateral force feedback and may be achieved via 
relatively simple, compact and low cost technology [2]. 
In this work we investigate the capability of tactile skin 
stretch feedback (SSF) to aid novice participants in 
perceiving the tension of simulated sutures during a 
‘grasp and pull’ task. The experiment is also conducted 
with force feedback (FF) and combinations of FF and 
SSF to permit comparative evaluation between feedback 
modalities and changes in perceptual capability. 

MATERIALS AND METHODS 

To investigate user ‘tension perception capability’ 
related to the SSF and FF haptic modalities under 
investigation, a novel multi-modal haptic interface (Fig. 
1) was combined with classic experimental methods of 
psychophysics. In the study, participants used the haptic 
interface to ‘grab and pull’ virtual suture threads, as if 
they were performing the last stages of tying a knot or 
pulling the thread through tissue. The study method was 
based on the classic psychophysics ‘ Method of Limits ’ 
protocol [3] and had ethical approval from the 
University of Bristol (application 4201). 



Fig. 1 : Novint Falcon and Tactile Display used in this study. 
By this method, in each trial, two virtual suture threads 
were presented to the participants, via display on a 
computer monitor. By moving the haptic interface with 
their finger, participants manipulated an on-screen 
cursor to ‘grab’ virtual objects attached to the end of the 
suture threads and laterally ‘pull’ the objects and threads 
towards themselves. The goal was to manipulate both 
threads and use the presented haptic feedback to 
determine the relative tension of a green, comparison 
thread to a red, reference thread. The participant 
selected greater or less using labelled buttons on a 
numeric pad. The red thread had a constant reference 
tension of 4N. The green thread began each trial either 
high (7, 7.5 or 8N) or low (0, 0.5 or IN) and increased 
or decreased accordingly in 0.5N increments. For more 
details on the Method of Limits protocol see [3]. 
Facilitating realistic two finger suture thread grasping 
motions and the tactile sensations of thin-diameter 
thread is technically difficult in a multi-modal (FF and 
SSF) haptic interface and beyond the scope of this 
investigation. As such, large wedge shaped objects (Fig. 
2) were implemented to facilitate grabbing the thread. 
The following description of objects, motions and forces 
is represented in Fig. 2. The threads were biased with 
different tensions T, which acted in opposition to the 
pulling force of the user F P . Motion of the object and 
thread, X (constrained to one dimension and direction) 
would only occur if F P > T. In addition, a downwards 
force F g was required to ‘grab’ the thread (via the 
connected wedge object). As a user exerted F P and F G , 
opposing reaction forces F PR and F GR , were calculated. 
Both reaction forces could be rendered by SSF and / or 
FF haptic feedback modalities. They were therefore felt 
by the user via the Tactile Display and / or Novint 
Falcon haptic device respectively. The tactile display 
features a 4x4 variable height tactile pin array mounted 
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Fig. 2: Simulated environment and tactile display. Reaction 
forces {F pr , F gr ) may be rendered by tactile display or Falcon 

on an actuated ‘lateral stage’ to permit the generation of 
skin stretch sensations both normal and lateral to the 
user’s finger pad. In this work lateral deformation is 
proportional to the suture pulling reaction force F PR , 
while the height of the pins, which are actuated as a 
whole in this study, are proportional to the grasping 
reaction force F GR (both via a 0.32mm/N ratio). Such 
indenting and shearing finger pad deformations are 
present in natural, every day interactions with real world 
objects in proportion to exerted forces [2]. Details of the 
tactile display are shown in Fig. 2, with more 
information available in [4]. F PR and F GR can also be 
represented, directly as kinesthetic forces, by the Novint 
Falcon haptic device. The tactile display is attached to 
the end effector of the Falcon (Fig. 1). To maintain 
constant position between the user’s finger pad and 
tactile stimulus, the second phalanx of the user’s 
dominant index finger is strapped to a finger rest with 
the distal phalanx held in place by a finger guide, 
available in various sizes. To keep the system 
lightweight and easy to move, the tactile display is 
remotely actuated via Bowden cables (Fig. 1). In order 
to compare the effect of different stimulus rendering, 
four feedback modalities (A-D) were presented to users: 



FF 

SSF 

A - Force 

F PR & F gr 

- 

B - Tactile 

- 

F PR & F gr 

C - Combined 

F p R & F gr 

F pr SlF gr 

D - Hybrid 

F G R 

F PR 


For each feedback modality, 8 (random and balanced) 
sequences of comparison stimuli (trials) were completed 
per participant. Familiarisation with comparison 
tensions of ON and 8N was completed before the study. 

RESULTS 

11 participants (9 male, ages 23-45) took part in the 
study and completed several feedback modalities each. 
Modes A and C both had 7 completed trials while 
modes B and D had 6. Combined psychometric 
functions [3] were generated for each modality by 
combining the results of all individual trials for that 
modality. These combined functions are presented in 
Fig. 3. In this figure, steeper gradients around 4N (the 
reference stimulus) indicate a greater sensitivity of 



Fig. 3 - Combined Pyschometric Functions. 


participants to a change in stimulus. It may be seen that 
the force feedback modalities A and C provide better 
perception of tension. B and D, which both used skin 
stretch feedback for representing F PR (and therefore 
tension) provided relatively reduced sensitivity, though 
D (the hybrid mode) is superior to B (SSF only). These 
results are numerically presented in the following table, 
which illustrates Point of Subjective Equality (PSE - 
where two tensions appear equal), Just Noticeable 
Difference (JND - the smallest change in tension that 
may be reliably detected) and Weber Fraction (stimulus 
change that can be reliably detected) for all modalities. 
Two tailed t-test comparisons for individual participants 
found statistically significant ( p<0.02 ) differences in 
JND and WF between modalities A & B and B & D. 
Force Tactile Combined Hybrid 

PSE(N) 4.032 4.592 4.176 4.184 

JND(N) 0.656 2.32 0.672 1.304 

WF (%) 16.4 55.8 16 1 8 32.6 


DISCUSSION 

The results indicated superior tension perception for 
force feedback (A) compared to skin stretch feedback 
(B). However, FF stability is presently too unreliable for 
commercial RMIS, despite considerable research efforts 
[2]. Combining SSF with FF (C) had little effect on FF 
only (A) results. SSF alone (B) led to relatively poor 
perception but the hybrid mode (D) significantly 
improved perception over B (p<0.02 ). In D, F PR , which 
related to tension, was displayed by SSF, while FF 
represented only the grasping force, F GR . More reliable 
grasping via FF improved SSF tension perception. 
Although our study used an abstracted component of 
suturing, it seems that SSF has potential, especially 
when compared to the current clinical standard of no 
haptics in RMIS. 
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INTRODUCTION 

Cleft palate is the most common congenital defect 
and requires surgical repair between 6-12 months of 
age to ensure proper speech development [1]. Surgical 
repair of a cleft palate is a challenging procedure due to 
the small confines of the infant oral cavity, and the 
delicate tissue handling and dissection required [2-4]. 
Access, visualization, instrument manipulation, 
ergonomic factors and teaching opportunities are all 
limited during the procedure. 

Previously described cleft palate simulators are very 
low fidelity and do not incorporate the majority of the 
relevant anatomy [2-4]. We propose developing a high- 
fidelity cleft palate phantom for training purposes. 

In addition, the surgical approaches used in cleft 
palate repair have remained unchanged over the last 
century [5]. The limited access, space and delicate 
tissue handling make cleft palate surgery an ideal 
candidate for robotic application. 

MATERIALS AND METHODS 

Cleft Palate Phantom Development 

A high-fidelity cleft palate phantom (patent pending 
62/100150 and 62/118306) was developed from a 
normal infant computed tomography (CT) scan. A 
three-dimensional (3D) representation of the palate, the 
relevant components of the cranial base and the oral 
cavity were segmented from the CT. A bony cleft was 
created using 3D modeling software. Computer 
animation software was utilized to develop the detailed 
musculature of the palate based off the anatomy in the 
literature [6] and discussion with expert cleft palate 
surgeons. The bony palate and molds of the muscles 
and oral cavity were created with a Makerbot® 
Replicator™ 2 (Makerbot Industries, New York, United 
States) 3D printer. The musculature and oral cavity 
were casted with silicone. The musculature, bony 
component and mucosa were assembled using medical 
grade adhesive. The model was developed with a 
cartridge that can be replaced after each use. 

Three staff surgeons and two fellows in Plastic and 
Reconstructive Surgery performed a von Langenbeck 


cleft palate repair using standard surgical instruments on 
the phantom model. 

Robotic Testing 

The feasibility of utilizing the da Vinci® Surgical 
System to perform a cleft palate repair using the 
phantom was conducted. Testing was performed using 
a first generation da Vinci ® Surgical System with 5 
mm Endowrist® instruments at the Canadian Surgical 
Technologies & Advanced Robotics Center (CSTAR) at 
the University of Western Ontario. 

RESULTS 
Phantom Model 

The complete cleft palate phantom model can be 
seen in Fig. 1 and Fig. 2 with the specific features 
described in Table 1. The three staff surgeons and two 
fellows reported the model replicated the critical 
components of cleft palate repair in a realistic fashion. 
The surgeons were successful in 1) creating lateral 
relaxing incisions, 2) elevating mucoperio steal flaps, 3) 
exposing the pedicle, 4) exposing the posterior cleft 
margin (Fig. 3A), 5) releasing the palatopharyngeus 
from the cleft margin (Fig. 3A), 6) release of the 
palatopharyngeus from anterior fibers of the tensor 
muscle, 7) exposure of the levator muscle travelling 
through the levator tunnel, 8) suturing the nasal mucosa 
(Fig. 3B), 9) suturing the levator muscles together, and 
10) suturing the oral mucosa together. 



Fig 1 . Perspective view of cleft palate phantom. 
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Table 1. Muscles, bony landmarks and oropharyngeal 
structures incorporated into cleft palate phantom. 


Muscles 

Bony 

Landmarks 

Oropharyngeal 

structures 

Superior 

Foramen magnum 

Soft palate mucosa 

constrictor 

Foramen lacerum 

Hard palate mucosa 

Levator veli 

Foramen ovale 

Oropharynx 

palatini 

Foramen spinosum 

mucosa 

Tensor veli 

Carotid canal 

Eustachian tube 

palatini 

Levator origin 


-Anterior fibers 

Tensor origin 


-Aponeurosis 

Hamulus 


Palatopharyngeus 

Greater palatine 


-Oral head 

foramen 


-Nasal head 



Palatoglossus 





Fig 2. Multi-layered silicone cleft palate phantom close-up 
surgical view with mouth retracted using the Dingman oral 
retractor. 



Fig. 3: Surgical views of performing cleft palate repair with 
phantom. A) Releasing palatopharyngeus (PP) from left cleft 
margin. B) Suturing nasal mucosa together. 


Robotic Testing 

An evaluation of the 3D workspace (3 cm by 3 cm 
aperture, depth 6.5 cm, suggested 5 mm Endowrist® 
instruments on the da Vinci ® Platform as a potential 
robotic solution. Close positioning of the patient side 
manipulators (PSM’s) allowed accessing both the soft 
and hard palate and a 30 degree endoscope positioned 
distal to the oral aperture resulted in excellent 
visualization of the hard and soft palate. 

Testing was successful in dissection of the mucosa 
from the hard and soft palate, elevation of 
mucoperiosteal flaps, releasing the musculature from 


the hard palate as well as suturing the nasal mucosa, 
muscles and oral mucosa together (Fig 4). However, 
frequent collisions between the PSM’s and oral aperture 
made the procedure challenging. 



Fig. 4: dissection of mucosal layer from right palatal 
musculature using phantom with 5 mm Endowrist® 
instruments and 30 degree endoscope. 


DISCUSSION 

We have successfully developed a highly realistic, 
functional cleft palate phantom that can be used as both 
a training tool and to test a robotic approach to cleft 
palate repair. Utilization of the da Vinci ® Surgical 
System for cleft palate repair using 5 mm Endowrist® 
instruments is feasible and robotic repair may have 
many advantages over standard instruments. Given the 
smaller size of the infant oral cavity compared to adults, 
the frequency of collisions was not surprising. 

Optimization of path planning, collision alerts, 
alternate tool configurations, and customized retractors 
would improve the efficiency and safety of the robotic 
procedure. 
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INTRODUCTION 

Lung cancer is one of the most profound current public 
health challenges, with more than 150,000 lives lost to it 
each year in the United States alone [1]. A major 
challenge is early diagnosis, since life expectancy drops 
precipitously as the tumor grows. While small nodules 
(which may or may not be cancerous) are being 
identified with increasing frequency due to better 
imaging and new guidelines that dramatically increase 
the frequency of screening, many cannot currently be 
biopsied. Surgeons have a low diagnostic yield rate 
(< 52%) with nodules less than 1.5 cm in size using 
hand-held needles [2]. Furthermore, traditional 
percutaneous approaches risk lung collapse, which is a 
serious complication that can even be fatal to patients 
with various co -morbidities. 

Transoral targeting for biopsy and therapy delivery 
is a better approach, since the lung’s outer wall is not 
punctured, reducing the risk of lung collapse. While a 
few academic and commercial systems exist for this 
purpose (e.g. Medtronic’s superDimension system 
http ://superdimension. com/) , they are typically limited 
to nodules that occur in or near the bronchial tree. We 
recently described a robotic system designed to exit the 
bronchial tree and steer through the soft tissue of the 
lung [3]. The system, shown in Fig. 1, combines three 
steerable surgical devices: a tendon-driven redundant 
mechanism (the bronchoscope), a concentric tube robot 
[4], and a flexure-based bevel tip steerable needle [5]. 
Concentric tube robots have been used to augment the 
dexterity of rigid [6] and flexible endoscopes [7] in 
prior work, but this is the first time all three steering 
technologies have been combined into one system. The 
intended workflow for our system is (1) deploy the 
bronchoscope manually to the desired location in the 
bronchi, (2) use a puncture mechanism [3] to make a 
small opening in the bronchial wall, (3) deploy the 
concentric tube robot through it into the soft tissue of 
the lung, and (4) deploy the bevel tip needle and steer it 
to the desired target under closed loop control [8]. 



Fig. 1 . Our transoral lung access system consists of three 
stages: a tendon- actuated bronchoscope, a concentric tube 
robot, and a steerable needle. The system was designed to 
target hard-to-reach nodules in the peripheral lung. 

In this paper we focus on improving step (3). Our 
objective is to provide a way for the concentric tube 
stage to use high curvatures, while minimizing tissue 
damage and deformation. We aim to achieve this via 
follow-the-leader deployment, in which the shaft of the 
device remains perfectly in the path traced out by the 
device’s tip as it advances through tissue. See [9] for a 
discussion of this as it relates to concentric tube robots. 
Achieving follow-the-leader deployment with our 
system is a challenge, because the long transmission 
lengths required for the tubes to pass through the 
bronchoscope are subject to substantial torsional windup 
as the curved tubes elastically interact. This can cause 
elastic instabilities and snap-through, in which the tubes 
suddenly snap from one configuration to another 
[10,11]. This makes a range of axial tube rotation angles 
inaccessible, unless tubes are only gradually curved, or 
are deployed such that their curved sections do not 
overlap when relative axial tube rotations are applied. 

Non-annular tube cross sections provide a way to 
achieve follow-the-leader deployment with higher tube 
curvatures, or at currently inaccessible relative angles. 
Non-annular tubes prevent relative rotation of the tubes 
with respect to one another, avoiding the snapping 
behavior caused by elastic instabilities. This idea was 
first developed in a collaboration between the last 
author and Philips, Inc. Yet to date, it has only been 
described conceptually in the patent literature [12]. 
Thus, the contribution of this paper is the first physical 
realization of non-annular concentric tubes. 
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Fig. 2. Our prototype concentric tube robot with elliptical 
cross sections. This design can achieve configurations that 
would be unstable for annular tubes, facilitating folio w-the- 
leader deployment with higher curvatures. 


maximum overlapped arc length of the precurved 
portions of these annular tubes for which stable opposed 
curvatures could be maintained would be 0.9 mm [10]. 
Furthermore, when the precurved sections of these 
annular tubes are fully overlapped and extended from 
the bronchoscope, the maximum stable rotation angle of 
the inner tube relative to the outer is 53°, and this 
requires a tube base angle of approximately 3083°, more 
than 8.5 revolutions! In contrast, the use of non-annular 
tubes enables higher curvatures and previously 
inaccessible relative tube angles (e.g. the 180° case 
shown in Fig. 2) to be employed. 


MATERIALS AND METHODS 

To create a set of non-annular tubes, we modified a set 
of annular tubes. We used an outer tube with an outside 
diameter (OD) of 1.60 mm and an inside diameter (ID) 
of 1.40 mm, and an inner tube with an OD of 1.16 mm 
and an ID of 0.99 mm. Mandrels made from nitinol wire 
(0.79 mm and 0.48 mm, respectively) were inserted into 
each tube, and a hot air gun was used to heat each tube 
to approximately 600° C. We then mechanically 
deformed the tube onto the mandrel with a set of pliers, 
yielding an elliptical cross section. We then precurved 
the tubes (curvature of 31.5 m" 1 , curved length ~55 
mm). Fig. 2 shows the tubes placed concentrically. The 
elliptical cross sections prevent relative tube rotation. 

RESULTS 

The complete system introduced in the previous section, 
and described in [3], is illustrated in Fig. 3. The 
enhancement to the system that is the main result of this 
paper is the prototype shown in Fig. 2. The elliptical 
cross sections enable tubes to deploy with curvatures 
opposed to one another (see Fig. 2), in configurations 
that would be unstable for annular tubes [10,11]. For 
example, annular tubes of the same dimensions, 
precurvatures, and curved lengths, with transmission 
lengths long enough to pass through a bronchoscope 
(e.g. 735 mm and 835 mm), would wind up torsionally 
and be unable to achieve this configuration. In fact, the 



Fig. 3. (a) Our complete system, in which the concentric 
tube robot and steerable needle pass through the 
bronchoscope port, (b) Photographs of phantom targeting 
experiments with the system, in which magnetic tracking 
feedback was used to steer the needle to targets [3]. 


CONCLUSION 

While non-annular tube cross sections would not be 
useful in contexts where the concentric tube robot is 
intended to act as a manipulator (see [4]), they can be a 
good option when one desires the concentric tube robot 
to act like a steerable needle. By locking the relative 
axial rotation of the tubes, elastic instability is 
prevented, and configurations inaccessible to annular 
tubes become possible. This is potentially useful in any 
procedure where the robot acts like a steerable needle. 
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INTRODUCTION 

Autonomous robot-assisted execution of repetitive 
surgical tasks has the potential to reduce fatigue, whilst 
improving surgical navigation, procedure times and 
workspace usage. The increased use of tele-surgical 
robots and development of open-source platforms offers 
new opportunities for autonomous navigation which has 
led to a growing interest in applying the learning -by- 
demonstration (LbD) framework used in robotics 
towards complete automation of repetitive tasks [1, 2]. 
While vision information has been used towards 
towards tool tracking and target detection for 
autonomous task execution [3], the combination of LbD 
with vision-based 3D landmark recognition and tool 
tracking towards adaptive trajectory transfer has not 
been explored. This paper proposes a LbD framework 
for automating surgical robotic execution of subtasks 
under stereo vision guidance, using the da Vinci 
Research Kit (DVRK). The approach consists of 
learning optimum motion trajectories for different 
subtasks of a suturing task, and using stereo vision for 
3D anatomical landmark recognition and needle 
tracking, to allow adaptation of learned trajectories to 
different clinical scenarios with moving camera views. 

MATERIALS AND METHODS 

This paper made use of the DVRK development 
platform provided by Intuitive Surgical with the 
software implementation based on the cisst/SAW 
libraries [4]. Two large needle driver tools were used at 
the patient side manipulator (Fig. 1). A custom-made 
PVA cryogel phantom (providing similar elastic and 
tensile properties as normal tissue) with customized 
built-in wounds (width=6mm) was made for these 
experiments. Motion models were learned from multiple 
experienced demonstrations (n=6,>10 simulator/surgical 
procedures) performing a running suture with three 
needle insertions. The suturing task is considered to 
consist of a set of subtasks to be modeled individually 
and executed by the robot automatically in a sequential 
order. However, the focus of this study is “Subtask 1: 
needle insertion through the wound with the right 
instrument”, which for this task can be considered the 
more dexterous subtask. This subtask is preceded by an 
initialization step involving moving the needle to the 
correct insertion point by the right instrument. 



Fig. 1 Experimental setup of the da Vinci system with the 
DVRK (a), needle tracking for moving the needle from initial 
position (b) to desired target insertion point (c), wound 
detection under different camera views (d, e). 

Each demonstration consisting of the 3D Cartesian 
positions and the rotations (represented as 4D 
quaternions) of the needle driver tip was manually 
segmented and trajectories from multiple 
demonstrations were temporally aligned using a 
combination of dynamic time warping and spherical 
linear interpolation. For learning the movement patterns 
and reproducing trajectories that capture the main 
features of the training data, this work makes use of 
Hidden Semi-markov Models (HSMM) [5], a form of 
explicit-duration Hidden Markov Model (HMM), where 
the self-transition probability of the standard HMM is 
replaced with a parametric model of the state duration, 
thereby providing a better balance between temporal 
and spatial constraints than a standard HMM. As a 
result, a smooth generalized form of the motion for the 
desired subtask was learned. 

The proposed vision algorithm is used for two main 
purposes: 1) wound detection for extracting the 
geometry and orientation of the wound under different 
anatomical variations/camera views 2) needle tracking 
in order to provide real-time 3D positioning of its tip. 

In order to move the needle from an arbitrary initial 
position to desired target insertion point (Fig. 1), after 
which automatic execution of the learned trajectories for 
Subtask 1 can be performed, the inverse kinematics of 
the robot is used. The transformation between the 
coordinate system at the base of the robot and at the 
stereo endoscopic camera was calculated offline through 
hand-eye calibration. 

Initially, the wound is detected in the stereo images 
through a semi-automatic algorithm that asks the user to 
provide two input points and geometrically models the 
wound with an ellipse, while a feature image based on 
the analysis of the eigenvalues of the Hessian matrix is 
computed to enhance the borders of the wound. Once 
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the 3D pose of the wound is estimated through 
triangulation, the needle is driven towards the desired 
target insertion point using the robot’s inverse 
kinematics. Needle tracking is achieved through an 
algorithm that detects small straight segments on a 
feature image and only considers segments close to the 
detected needle in previous frames and with similar 
orientation. The segments are combined to create a 
continuous curve that represents the needle shape in the 
current frame. Using these results the 3D needle tip 
position is estimated through triangulation. 

RESULTS 

Fig. 2 shows the running suture task (a) as well as the 
HSMM structure for Subtask 1, as learned from 
multiple experienced demonstrations, and the smooth 
generalized version of the motion extracted (b), with 
different colors corresponding to the different states of 
the HSMM. The optimum number of states was selected 
using the Bayesian information criterion. 



Fig. 2 The running suture task (a) and the learned motion for 
“Subtask 1” (b) as reproduced from multiple demonstrations 
using the HSMM. The ellipses correspond to the covariances 
for each state of the HSMM. 


The proposed method has been validated by automating 
the needle initialization towards the target insertion 
point, using the vision system, followed by automated 
playback of the learned trajectory for Subtask 1. The 
whole execution was considered successful if the needle 
tip passed through the wound, and exited on the other 
side, enough to be grasped by the left instrument. The 
automation has been tested with the wound under 
different camera views, ranging from a viewing angle of 
-10 degrees to +10 degrees with respect to the camera, 
as well as different initial poses and distances of the 
needle with respect to the wound. The algorithm vas 
validated multiple times (n=7) under these different 
initial conditions. In each case the wound detection 
algorithm estimates the 3D pose of the anatomy which 
was used to drive the needle towards the desired target 
insertion point. Fig. 3 shows examples of the needle 
driver tip trajectory for automated initialization of the 
needle at the desired insertion point in the robot base 
coordinate system (depicted in orange), followed by 
automated playback of the learned motion for Subtask 1 
(depicted in blue). The initialization step is sensitive to 
the accuracy of the wound detection, the robot 
kinematics, and the hand-eye calibration. The results 
depict successful and smooth execution of the desired 
motion under different initial conditions. 






i 
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Fig. 3 Examples of the needle driver tip trajectory for 
automated initialization of the needle at the desired insertion 
point (orange) followed by automated playback of the learned 
motion for the first subtask (blue) under different initial 
conditions in the robot base coordinate system [mm] . 


DISCUSSION 

This paper proposes a LbD approach, using the DVRK 
framework, for learning and automating motion 
trajectories for different sub tasks of a suturing task 
under vision guidance. By using stereo vision for 3D 
perception, anatomical landmark detection and needle 
tracking, the approach addresses different clinical 
scenarios under varying initial conditions and camera 
views, thereby allowing for subject- specific variability 
and tissue deformation. Future work will further extend 
the framework for adapting the learned trajectories to 
more complex anatomical variations by combining the 
needle tracking with visual servoing, to further 
overcome the inherent errors in the needle initialization. 
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INTRODUCTION 

Minimally invasive surgery (MIS) brings multiple 
benefits to patients, such as shorter hospital stay, lower 
incidences of intra- and post-operative complications, and 
better cosmesis. The growing popularity of MIS leads to 
a vast adoption of endoscopy devices, of which the 
market is $28.2 billion in 2013 and is expected to reach 
$37.9 billion by 2018 [1]. Existing endoscopes fall into 
three categories, i.e. the rigid endoscope (RE), the 
flexible endoscope (FE) and the capsule endoscope (CE). 
Among them, REs are most frequently used in the 
minimally invasive cardiac surgery (MICS), in which the 
endoscope enters via an incision on the chest wall much 
smaller than that used in conventional cardiac surgery. 
Since the tip of the RE cannot be steered, it is difficult to 
view the entire surgical site with one RE. Also, some 
regions cannot be viewed due to the blockage of tissues. 
To solve these drawbacks, it is desired that the tip be 
flexible and can be steered. Hence, a FE with a rigid shaft 
and a steerable section is in favor, e.g. Endoeye Flex by 
Olympus [2]. However, in this type of FE, the length of 
the bending section is fixed. This results in the flexible 
section sweeping a large area to view from side to side. 
In cardiac surgery, the sweeping action is often limited 
by the heart and surrounding vital structures, and hence 
the field of view remains restricted. To solve these 
problems, a new cardioscope design is needed. 

The biomimetic wire-driven mechanism (WDM) is 
well suited to developing flexible devices [3-6]. It 
comprises of a flexible backbone and a set of controlling 
tendons/wires. Its motion is similar to that of the flexible 
endoscope. However, by using a constraint, bending of 
the flexible section is controllable and the workspace is 
expanded [7, 8]. This can increase the scope of vision. 

MATERIALS AND METHODS 

In this paper, we present a novel flexible endoscope 
which is well suited to MICS. It is named ‘the 
Cardioscope’. It is composed of a handle, a rigid shaft, a 
flexible section and the camera. The flexible section is 
composed of an elastic tube, a number of spacing discs 
and four tendons. The spacing discs are evenly 
distributed along the elastic tube. The tendons pass 
through the pinholes on the spacing discs and are fixed at 
the end of the flexible section. The other end of the 
tendons is attached to the tendon driving module, which 
is inside the handle. The flexible section is connected to 
the handle by the rigid shaft, which contains two 
concentric tubes, one is the shaft and the other is the 


overtube as shown in Fig. 1(a). The overtube can slide 
along the shaft by a sliding knob in the handle. It serves 
as a constraint to the flexible section. By adjusting the 
position of the overtube, the flexible section is 
constrained so that only the part outside the overtube can 
bend. The flexible section can bend in all directions by 
controlling four tendons and hence, the angulation of the 
bending section is also steerable. The bending motion is 
shown in Fig. l(b)-(d): Fig. 1(b) shows the regular 
bending motion, in which the length of the bending 
section is fixed; Fig. 1 (c) shows the bending motion, in 
which the length of the bending section is controlled. The 
four positions (i-iv) of the constraint and the 
corresponding bending are given. 



Fig. 1 Design of the flexible cardioscope: (a) the system 
overview; (b) bending of the flexible section - by controlling 
the bending angle (traditional way); (c) bending of the flexible 
section - by controlling the length of the bending section; (d) 
2D bending of the flexible section. 

A prototype is developed and ex-vivo tests are 
carried out to evaluate the design. The experimental setup 
is shown in Fig. 2. In the test, a pig heart is suspended by 
sutures and the endoscopes enter from the side holes, 
which mimic the incisions in the MICS. Two sets of tests 
are performed. In the first test, a mitral valve ring is 
placed in the pig heart and three different endoscopes are 
used to view the ring from a side pinhole mimicking the 
incision in the MISC. The first is a 0° RE and the second 
is a 30° RE, both from Karl Storz; the third is ‘the 
Cardioscope’. Three tasks are assigned to the endoscopes: 
task A - find the ring; task B view the full ring; task C - 
view the full ring in a single view. In the second test, the 
Cardioscope is used to explore the whole heart. 
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Fig. 2 The ex-vivo experiment setup and cardioscope in one of 
the bending configurations. 


RESULTS 

Results of the first test are summarized in Table 1. In the 
first test, all three endoscopes can find the ring 
successfully. However, for the 0° RE and 30° RE, due to 
the restricted scope of vision, they cannot view the full 
ring. The best views of the two REs are shown in Fig. 3 
(a) and Fig. 3 (b). The Cardioscope is tested in three 
configurations: the first is straight configuration; the 
second is bending with long bending section; the third is 
bending with short bending section. At the straight 
configuration, the best view of the Cardioscope is shown 
in Fig. 3 (c). It successfully finds the ring, however, the 
scope of vision is much smaller than the two commercial 
REs. The quality of the image is poorer. Considering this 
is the first prototype, the quality of image is not a key 
performance index. When the bending section is long, the 
Cardioscope successfully find the ring and can view the 
full ring. The views obtained at different bending length 
configurations are shown in Fig 3. 


Table 1 - Summary of the results of test 1 




Task A 

Task B 

Task C 

Olympus 0° RE 

Yes 

No 

No 

Karl Storz 30 

° RE 

Yes 

No 

No 


Straight 

Yes 

No 

No 

Cardioscope 

long bending length 

Yes 

Yes 

No 


short bending length 

Yes 

Yes 

Yes 



(a) (b) (c) 



(d) (e) (f) 


Fig. 3 Results of test 1 : (a) best view of the 0° RE; (b) best view 
of the 30° RE; (c) best view of the cardioscope at straight 
configuration; (d) and (e) first and second view of the 
cardioscope with long bending section; (f) view of the 
cardioscope with short bending section. 

In the second test, the pig heart is placed in a 
position similar to that in the MICS. The Cardioscope is 
controlled to explore the entire heart. Twelve snapshots 
are shown in Fig. 4. In the test, the surgeon successfully 
identifies the mitral valve, the aortic valve, the aorta, the 
pulmonary artery, the wall of the ventricles, and even the 
apex of the heart from a retrograde bending position. 



Fig. 4 Results of test 2: Cardioscope is controlled to explore the 
entire heart posed in the surgical position. 

DISCUSSION 

The presented Cardioscope shows the ability to provide 
much wider scope of vision than traditional endoscopes. 
By controlling the length of bending, the scope of vision 
is also tunable. In the test, the Cardioscope successfully 
explores the full heart through a single hole, which shows 
the design is promising. Although designed for MICS, 
the Cardioscope can also be applied to other MIS, such as 
laparoscopy, and neurosurgery. 
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INTRODUCTION 

A novel, small, and lightweight patient-mounted MRI- 
compatible robot has been developed for MRI image- 
guided interventions. Arthrography, biopsy, facet joint 
injection, and other percutaneous interventions which 
typically require radiation-based imaging guidance, 
could benefit from using this new robotic system. In 
typical manual image-guided interventions, the 
physician guides the needle using cross-sectional 
images to reach the desired position. Traditional needle 
manipulation often requires multiple passes to reach the 
target which prolongs the procedures and exposes the 
patient to undesired ionizing radiation. MRI-compatible 
robots may be enabling for radiation free procedures, 
which is especially important for pediatrics. 

The envisioned use of our prototype robot is as follows: 
after placing and securing the robot on the patient, and 
performing robot to scanner coordinate system 
registration, the radiologist will select a target and entry 
point from the MR images; the robot will then align a 
needle guide along this trajectory; and the radiologist 
will then insert the needle to the desired depth. The 
system configuration is shown in Fig. 1 . 

A few research groups have reported related work in the 
field of patient-mounted robots for percutaneous 
interventions. Walsh et al. [1] developed a patient- 
mounted robot called Robopsy. This was a system with 
4 degree of freedom (DOF) for needle holding, 
guidance, and insertion. A 2 DOF MRI coil-mounted 
robotic positioner for cryoablation was developed in [2]. 
The needle entry point had to be found and marked 
before attaching the robot to the patient, since this robot 



Fig 1 System configuration. 


did not have translational DOFs. Bricault et al [3] 
developed a light (lKg) and compact (15x23cm) robot 
with 5 DOFs to perform abdominal and thoracic 
punctures under CT or MRI guidance for diagnostic or 
therapeutic purposes. Song et al. developed a 2 DOF 
MRI-compatible double ring remote center of motion 
(RCM) mechanism for MRI-guided liver interventions. 
This device was a passive mechanism which was 
manually operated [4] . 

In this paper, we introduce a second version of our 
prototype robot [5]. An overview of control architecture 
and hardware is provided. The results of a recent 
accuracy study are reported. 

MECHANICAL SYSTEM 

Fig. 2 shows the new design and prototype of our 
patient-mounted robot created using an Objet 500 rapid 
prototyping machine. This robot is a 4 DOF robot, 2 for 
needle translation (rl and dl in Fig. 2), and 2 for needle 
rotation (r3 and r4). Some modifications were done 
from the previous design [5] to reduce the friction and 
backlash, and to increase the accuracy of the system. 
MRI-compatible piezo-motors (Piezo LEGS® Upsala, 
Sweden), and MRI-compatible Encoder: E8P OEM 
Optical Kit Encoder, 512 CPR, 1/8 bore, differential, 
hole in (US digital, Vancouver, Washington, USA) were 
used for robot’s joint actuation and measurement. A full 
ceramic ball bearing (ID=90 mm, Boca Bearing Inc., FL, 
U.S.A) was used for the base rotation to eliminate the 
backlash and friction associated with a sliding rotary 
bearing. 


Fig. 2 Prototype of our 4DOF robot. 

SYSTEM IMPLEMENTATION 

As shown in Fig. 1 the control system consists of four 
components: 1) robot controller, 2) robot/scanner 
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registration software, 3) user interface software, and 4) a 
communication channel to transfer MRI DICOM 
images and control signals between the scanner room 
and the control room. As shown in Fig. 3 the robot 
controller is located inside the MRI room, while the user 
interface software and image registration software run 
on a computer located in the control room. All cables 
between the scanner room and the control room are 
passed through the patch panel. Fiber optic cables were 
used to minimize the RF noise inside the MRI room. 
Preliminary tests in the MRI environment showed that 
the robot introduces some noise when activated. 
However, we do not anticipate this to be a crucial 
concern as MRI images will only be acquired when the 
motors are in an ‘OFF’ state. As part of our ongoing 
work, we are investigating quantitative metrics of image 
distortion under various use scenarios. 



y axis 

Fig. 4 Position accuracy study results for a quarter of the 
workspace. 

The root mean square (RMS) for rigid body definition 
was between 0.08 to 0.20mm during these experiments. 


ACCURACY STUDY 

Goal: To investigate needle positioning and orienting 
accuracy. 

Experiment Setup: To track the position and 
orientation of the robot’s needle guide, a rigid body, as 
shown in Fig. 3, was attached to the needle guide. This 
rigid body consisted of 8 active embedded markers. An 
optical tracker (Certus, NDI, Waterloo, ON, Canada) 
was used to track the rigid body (See Fig. 3). The 
manufacturer’s stated accuracy is 0.1 mm. The optical 
tracker was placed 2m away from the markers. 

Experiment Workflow and Data Collection : The robot 
was moved in the xy plane in 5 mm increments along 
both axes, and the position of the rigid body recorded. 
The robot was commanded to move to 48 different 
positions throughout the workspace. The orientation 
accuracy study was performed with 1 degree increments 
along both orientational DOFs. At each location, the 
position and orientation of the rigid body was recorded 
for 1 second at a sample rate of 120Hz, and averaged. 



Fig. 3 Sytem setup; yellow dash line shows which part of the 
setup is placed in MRI control room. Optical tracker was used 
for accuracy study in the lab environment. 


Results : The maximal position and orientation errors 
for the robot were ±0.6 mm and ±0.5 degree. Fig. 4 
shows the position accuracy study results for a quarter 
of the robot’s workspace. Based on clinician feedback, 
overall needle placement accuracy under 3 mm is 
considered sufficient for a conventional shoulder 
arthrography procedure. For the intended clinical 
procedure, this robot should be accurate enough. 

DISCUSSION 

This paper presented the second prototype of our 
patient-mounted MRI-compatible robot. The robot is 
intended to provide better targeting ability, improve the 
clinical workflow, and allow better access within the 
MRI scanner bore for needle-based percutaneous 
interventions. The robotic system architecture and an 
accuracy study were presented. In the next phase of our 
work, we will continue with MRI imaging studies, and 
evaluate the clinical accuracy in a phantom study in the 
MRI suite. 
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INTRODUCTION 

Virtual fixtures, or active constraints, are a controls 
concept that enable robots to intelligently collaborate 
with human users to complete tasks that are too 
complex for the robot to complete autonomously and 
too challenging for the user to complete unassisted. The 
body of work on dynamic active constraints is not 
extensive, but combinations of guidance and forbidden- 
region dynamic virtual fixtures for teleoperation have 
been studied. [1] Fixture evaluation methods in previous 
work include potential fields determined from pre- 
operative images, predicted and current positions of the 
environment surface location, and streaming point 
clouds. Some of these do not allow the slave to 
penetrate the forbidden region, which is impractical in 
surgical situations. This work proposes an impedance- 
controlled regional 3D dynamic virtual fixture generated 
based on dynamics of the heart motion. This virtual 
fixture has been added to the heart motion compensation 
system implemented on the da Vinci Research Kit 
(dVRK) described in [2] . In this work it was noted that 
while a user was operating on a virtually stabilized 
heart, the dynamics of the heart target were not 
accurately perceived. This resulted in additional, 
inadvertent perforations. To augment the surgeon’s 
perception, we propose a novel non-continuous, strobe- 
like virtual fixture whose enforcement is conditional 
upon the kinematics of a beating heart’s trajectory. 

MATERIALS AND METHODS 

A. Virtual Stabilization System overview 

The heart motion compensation system from [2] was 
implemented on the dVRK at the Collaborative Haptics 
and Robotics in Medicine Laboratory at Stanford 
Univeristy (Fig 1). In the heart motion compensation 





Fig. 1 dVRK set up for virtual fixture experiments for beating 
heart applications at Stanford University. 


system, the Patient Side Manipulators (PSMs) are 
programmed to follow the trajectory of a point on the 
heart surface, with superimposed teleoperation 
commands issued from the Master Tool Manipulators 
(MTMs). This results in a virtual stabilization of the 
heart motion from the perspective of the surgeon. The 
3D printed plastic heart target is controlled by one PSM 
while the surgeon controls the other PSM. The camera 
is attached to the plastic target at a fixed location 
relative to the simulated heart surface, emulating perfect 
camera motion tracking. 

B. Virtual Fixture Design 

In previous user studies for the heart motion 
compensation, it was difficult to see when the heart 
surface is accelerating upwards. This strobe fixture thus 
provides pulsing haptic feedback, active during periods 
of high acceleration in the heart trajectory. A 2 nd order 
Butterworth Filter is used to calculate the z-acceleration, 
a z , of the heart using joint velocity readings from the 
position sensors in the PSMs. It was experimentally 
determined that, for the trajectory (as in [2]) with a z- 
displacement of 11mm at 1.5Hz, the virtual fixture 
strobe pulse commences when a z > 0.39m/s 2 and 
terminates when acceleration falls below 
a z < -0.145m/s 2 . The force feedback on the master is 
displayed only in the positive z direction. The 
magnitude of the feedback is calculated using the 
relative position of the slave tool tip and a proxy 
following the heart surface (similar to the concept in [3]). 
The tool tip and proxy follow an elastic linkage model: 

~ z heart ~ z tool tip 

r _ f VF scale * (^z + V ^margin) if > 0 

M ™ z lo otherwise 

A virtual margin of 5mm is added to the surface of the 
heart so the imperceptibly low F MTMz occurs before 
collision between the tool tip and the heart surface. The 
dynamics and inertia of the MTMs guarantee a smooth 
transition across the F MTMz = 0 threshold. VF scale « 
150 ; A z is usually only a few millimeters so large 
scaling values are needed to generate perceptible haptic 
feedback. The torque applied to the MTM motors to 
display the haptic feedback is clipped to 1.0 Nm to 
avoid instabilities. The virtual fixture control loop runs 
at a frequency of 1kHz. 

C. User Studies 

Seven right handed subjects completed two one-handed 
tasks in an IRB -approved study with informed consent: 
drawing a circle (relevant to the standard Fundamentals 
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of Laparoscopy circle cutting task), and a simulated 
suturing task (piercing printed targets with a needle) 
(Fig 2). Heart motion compensation was active for both. 



Fig. 2 Tasks completed for user studies. Left: drawing a circle. 
Right: simulated suturing. 


For drawing a circle, a 3D printed adapter was used to 
attach the marker coaxially to the instrument shaft. For 
suturing, a straight needle and 6 pairs of dots were used. 
Each dot is a circle 1mm in diameter and the needle is 
0.5mm in diameter; high accuracy (similar to actual 
anastomosis) is required for this task. A target is “hit” if 
the needle perforation is completely within the printed 
dot, “missed” if the perforation is completely outside 
the printed dot, and “almost” otherwise. The subjects 
had varying levels of expertise with the robot and haptic 
devices. Each task was performed under four 
conditions; the subjects completed practice and three 
test trials for each. The conditions were (Cl) no 
feedback; (C2) basic moving wall virtual fixture: 
feedback is calculated using the elastic linkage model 
(described in B). This has been implemented in previous 
work [4], and is the baseline for comparing the effect of 
the non-continuity of the proposed virtual fixture; (C3) 
haptic pulse feedback: the pulse is active during high 
acceleration periods, irrespective of the relative 
positions of the tool tip and heart surface (C4) pulsing 
moving wall virtual fixture: based off C2 with 
additional dependencies on the dynamics of the target. 

RESULTS 

The performance of the subjects under the various 
conditions was evaluated based on four indicators: 

i. Efficiency - time of completion 

ii. Accuracy - how many times the marker tip left the 
surface (circle task); how many hit/missed/almost 
targets (suture task) 

iii. Damage - the “collision extent” or time under the 
surface multiplied by the distance under the surface 

iv. Safer Motion - to evaluate the efficacy of the non- 
continuity, we compare the motion of the 
teleoperated slave during the high-risk periods of the 
trajectory to the motion during lower-risk periods. 

The results shown in Fig 3 (except for the circle results 
in row II) depict the average over the 7 users. Efficiency 
and accuracy measures are normalized relative to the 
results for Cl, showing the effect of the feedback. 

DISCUSSION 

Based on the efficiency results, C2 was comparable to 
C4. However in the accuracy results for the circle task, 
C4 was the only condition with improved performance 
over Cl. The collision extent results show the benefits 
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Fig. 3 Results for user studies. Left column shows circle task 
results, right column shows suture task results. Row I shows 
efficiency; row II shows accuracy; row III shows damage; and 
row IV shows safe motion results (green: low-risk periods, 
red: high-risk periods). 

of virtual fixtures, where all conditions with haptic 
feedback surpassed the results from Cl. The ranking of 
the feedback conditions varied depending on the task. 
The advantage of the non-continuity is seen in the safe 
motion results, where in C3 and C4 the users performed 
a greater portion of the total displacement during the 
low-risk periods of the heart trajectory. There was high 
variance in performance between the different users; the 
tasks may have been too difficult for the skill levels of 
the participants, particularly the suturing task. Overall, 
the results show that the pulsing virtual fixture succeeds 
in relaying to the user the ideal times at which they 
should perform their tasks. The pulsing virtual fixture 
also improves completion time and accuracy of simpler 
tasks such as circle drawing. 
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INTRODUCTION 

Since the introduction of passive commercial capsule 
endoscopes, researchers have been pursuing methods to 
control and localize these devices, many utilizing 
magnetic fields [1, 2]. An advantage of magnetics is the 
ability to both actuate and localize using the same 
technology. Prior work from our group [3] developed a 
method to actuate screw-type magnetic capsule 
endoscopes in the intestines using a single rotating 
magnetic dipole located at any position with respect to 
the capsule. This paper presents a companion 
localization method that uses the same rotating dipole 
field for full 6-D pose estimation of a capsule 
endoscope embedded with a small permanet magnet and 
an array of magnetic-field sensors. Although several 
magnetic localization algorithms have been previously 
published, many are not compatible with magnetic 
actuation [4, 5]. Those that are require the addition of an 
accelerometer [6, 7], need a priori knowledge of the 
capsule’s orientation [7], provide only 3-D information 
[6], or must manipulate the position of the external 
magnetic source during localization [8, 9]. Kim et al. 
presented an iterative method for use with rotating 
magnetic fields, but the method contains errors [10]. 
Our proposed algorithm is less sensitive to data 
synchronization issues and sensor noise than our 
previous non-iterative method [11] because the data 
from the magnetic sensors is incorporated independently 
(rather than first using sensor data to estimate the field 
at the center of the capsule’s magnet), and the full pose 
is solved simultaneously (instead of position and 
orientation sequentially). 

MATERIALS AND METHODS 

Localization is performed relative to the robot’s frame, 
O r , which we place at the location of the rotating dipole 
(i.e., the center of the actuator magnet). The capsule’s 
coordinate frame origin, O c , which resides at the center 
of the capsule’s magnet, is described by the vector Pq 
(Fig. 1). The rotation matrix, R , describes the capsule’s 
coordinate frame relative to the robot’s. Our goal is to 
solve for Pq and R. 

This method assumes the capsule is free to move, but 
that the dipole-field rotation is well beyond the “step- 
out” frequency, where the field is rotating too quickly 
for the capsule to rotate synchronously, such that we can 
assume no net motion and decouple the localization and 
actuation of the capsule. In our setup, the field source is 

This material is based upon work supported by the National Science 
Foundation under Grant No. 0952718. 



Fig. 1 Experimental setup with a spherical-actuator-magnet 
manipulator [12], mounted on the end of a 6 -DOF robot. A 
prototype capsule freely rotates and slides in a clear, acrylic, 
lubricated tube. The inset depicts the capsule’s sensor layout 
[11]. Each sensor is labeled with the field component it 
measures and its offset, 5, in mm, from the internal magnet’s 
center. The grey sensors are not visible from this angle, but are 
at symmetric offsets. 

a spherical permanent magnet, which is accurately 
modeled by the point-dipole equation [3]: 

B(p) = — , ( 3 pp T -l)M ( 1 ) 

4*||p|| 3 ’ 

where B(p) is the field at location p, p is the unit- 
normalized p vector, M is the dipole moment of the 
magnet, I is the identity matrix, and Pq is the 
permeability of free space. 

We previously developed an array of six Hall -effect 
sensors to surround the magnet inside the capsule to 
estimate the applied field at its location [11] (Fig. 1). 
The sensors are placed at known offsets, S h as noted in 
Fig. 1. The position vector, P/, describing sensor i in the 
robot frame, is p, = p c + R6 r The scalar magnetic- 
field projection measured at each sensor is: 

B/iPi) = sj R t (3p,pJ - I || Pi || 2 ) M (2) 

4*11 P/P 

where s t is the sensor’s measurement axis in the capsule 
frame, which is known. The nonlinear least-squares 
Levenberg-Marquardt algorithm in MATLAB was 
implemented to estimate the capsule’s pose by 
comparing the sensor data to the estimated field at each 
of the sensor positions using Eq. 2. 

The spherical-actuator-magnet manipulator (Fig. 1) 
uses three omniwheels to provide a singularity -free 
rotating dipole field [12]. It is mounted on the end of a 
6-DOF robot. A prototype capsule, 50.5 mm in length 
and 25 mm in diameter, was placed in a lubricated 
acrylic tube in which it is free to move, at a known 
position and orientation with accuracy of 2 mm and 10°, 
respectively, based on visual inspection. 
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The actuator magnet was rotated at 2 Hz around the 
x, y, and z axes, which resulted in step-out, with the 
actuator magnet’s position held stationary. Data was 
collected at 100 Hz and wirelessly transmitted, in 
batches, at 20 Hz to a computer. Two rotations about 
each axis were combined, for a total of 3 s of data (1800 
sensor measurements) to localize at a given position. 
For positions to test, we chose two orthogonal planes as 
depicted in Fig. 2. 

RESULTS 

The localization results are shown in Fig. 2. The total 
position error in mm and the orientation error in degrees 
are shown next to their corresponding position. The 
orientation error is in terms of the angle-axis 
representation. The average error across all points was 
6.9 mm and 10.7°, the average computation time for the 
least-squares algorithm was 3.4 seconds. As expected, 
the position error tends to increase as the magnet is 
moved farther away from the capsule, due to reduced 
signal-to-noise. For both planes, the radial positions 
(along the r axis) result in more accurate position 
estimates than the axial positions (along the -x axis). We 
do not observe any clear trends in orientation error, but 
this could be partially due to our ground-truth 
orientation error having comparable values. Note that 
the two axial positions are the same in planes A and B , 
so the differences in the localization estimates give an 
indication of the variance that can be expected. This 
method assumes that the capsule has no net motion; 
some of the error is due to our experiments not 
respecting this assumption, as would be true in practice. 

DISCUSSION 

The accuracy obtained is likely to be sufficient for use 
with our previously published magnetic-actuation 
method [13]. It performs comparable to other 6-D 
localization methods; [7] has an average error of 5 mm 
in a slightly smaller 15 cm sphere. This localization 
method is not capable of localizing a synchronously 
rotating capsule because of the assumption that the 
capsule has no net motion. However, a simple control 
scheme involves pausing actuation at periodic intervals, 
and either increasing the rotation speed of the dipole 
field or rotating it around an orthogonal axis, either of 
which will cause the capsule to stop rotating 
synchronously, and after collecting a few rotations of 
data, continue propelling the capsule in a desired 
manner. No additional movement of the external dipole 
source is necessary. For the reported experiments, we 
arbitrarily chose to use two rotations about three 
orthogonal axes. Further investigation needs to be done 
to determine the relationship between the amount of 
data collected and the resulting accuracy. Subsequent to 
the reported experiments, we built a smaller version of 
the prototype capsule that is 36 mm in length and 13.5 
mm in diameter. Although it is conceivable that the 
hardware size could be further reduced, we believe this 
will be sufficient for clinically realistic trials in future 
work. 



-x | m m | 

Fig. 2 The inset shows the two orthogonal planes tested. The 
capsule’s z axis is aligned with its magnet’s dipole axis m. On 
each plane, dots show positions tested. The position (mm) and 
orientation errors are shown next to the corresponding dot. 

The radial distance r refers to the z axis in plane A and the -y 
axis is plane B. 
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ABSTRACT 

To date, applications of Google Glass in the operating 
theatre, including video streaming for training and 
telementoring [1], can be characterised by the outward 
flow of information from the device. No precedents 
have been set where the platform acts as the delivery 
mechanism for patient-specific 3D images, providing 
anatomical guidance at key stages during surgical 
procedures. In laparoscopic and robotic surgery, the 
natural destination for such information is the existing 
display device. Most operating theatres for open 
interventions have fixed imaging displays, but there is a 
compelling case for equipping the surgeon with a 
personal navigation feed, manipulatable in real time, 
always within the field of view, delivered unobtrusively 
through a wearable device, so that attention is never 
diverted away from the surgical field. 



Fig. 1 (clockwise from top) First deployment of Google Glass 
for surgical guidance; Myo gesture control armband; open 
palm gesture used to engage orientation control. 


Under an existing ethical protocol (REC reference 
07/Q0703/24), informed and written consent was 
obtained from a 65 year old male patient requiring open 


aortic and bilateral common iliac aneurysm repair. 
Preoperative contrast-enhanced CT imaging was 
segmented manually to demonstrate the spatial 
relationship of each ureter to the dilated aneurysm sacs. 
These mesh-based models were complemented by real- 
time volumetric renderings of the raw DICOM data, 
provided by the Fovia HDVR® engine. Powerful 
hardware facilitated creation of a video stream on-the- 
fly. This underwent GPU-based H.264 encoding and 
was served as a RTSP stream to the Google Glass client 
over a local wireless network. 



Fig. 2 (clockwise from bottom) Vessel preparation prior to 
placement of graft; segmented preoperative CT scan showing 
distorted path of ureters; volumetric rendering of main vessels 
demonstrating tortuosity and internal iliac origins. 

Manipulation of the image viewpoint and orientation 
was made possible through a wireless Myo gesture 
control armband worn underneath surgical scrubs. In 
tandem with a 9-axis inertial measurement unit, 
proprietary electromyography (EMG) sensors measured 
electrical activity from forearm muscles, detecting up to 
five command poses. Auxiliary control was also 
provided via an existing iPad gesture interface [2] 
managed by the assistant. Figure 2 (top) illustrates 
typical images viewed in the display. 
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INTRODUCTION 

Robot-assisted minimally invasive surgery is a research 
area of continuous evolution. A number of continuum 
robotic manipulator has been designed to overcome the 
limitation of traditional rigid instruments. The high 
number of degrees of freedom of these kinds of robots 
provides superior redundancy and allows reaching 
further inside the abdominal cavity of a patient avoiding 
organs in the way [1], [2]. This capability has been 
proven to consistently increase the workspace inside the 
patient body, and, hence, to enhance the efficiency at 
which the surgeon can conduct minimally invasive 
surgery (MIS) tasks [3] with a great potential of 
improving the operation’s outcome for the benefit of the 
patient. 

STIFF-FLOP is an EU-FP7 project which aims to create 
novel soft continuum manipulators taking inspiration 
from the octopus arm [4] [5]. The soft manipulation 
concept proposed here is based on an antagonistic 
actuation mechanism combining extrinsic (tendon- 
driven) and intrinsic (pneumatic) actuation [6]. This 
principle is used by nature in various ways, e.g., in the 
octopus arms which have longitudinal and transversal 
muscles. Activating both sets of muscles increases the 
arm’s stiffness [7]. A similar behaviour is achieved with 
the hybrid manipulation principle presented here. By 
fusing the advantages of both actuation types, the 
stiffness of the soft manipulator can be varied. The 
manipulator is able: 

• to change its configuration depending on the 
number of actuated tendons (see Figures 1), 

• to vary its length by changing the air pressure in 
the robot’s main body, and 

• to vary its stiffness and configuration through the 
controlled interplay of applied pressure and tendon 
displacement. 

The combination of intrinsic and extrinsic actuation 
principles has been investigated in a number of works 
[6]. An example of hybrid actuation is presented in [8] 
where a combination of pneumatic and electro- 
mechanical actuation is described proving an enhanced 
manipulation capability. The idea of a tendon-driven 
actuation combined with a pneumatic one has been 
investigated in [2], where each robot segment is 
composed of a pres suri sable chamber divided from the 
adjoining ones by an aluminium disk to which a set of 
tendons are attached. In our earlier paper [6], we 


presented a novel model tendon-driven and pneumatic- 
actuated soft manipulator. 



(a) (b) (d) 

Fig. 1 (a) and (b) Examples of achievable configurations of 
the manipulator, (c) and (d) Application in minimally invasive 
surgery . 

This paper proposes an evolution of our previously 
presented manipulator with a novel design conceived 
specifically for robotic surgery applications (see 
Figure 1(c) and (d)). Section II presents the overall bio- 
inspired system and its actuation architecture. Tendons 
are integrated to tele-operate each section, pressure 
control and the combination of tendon and pressure 
actuation is implemented to change the length and 
stiffness of the robot working antagonistically as in the 
octopus muscles. The experiments conducted are 
described along with results in Section III. Section IV 
summarises the achievements of this paper and presents 
future works. 

MATERIALS AND METHODS 

Figure 2 (a) shows the conceptual system architecture of 
our hybrid-actuated manipulator. 3D and section views 
of the manipulator are shown in Figures 2(b) and (c), 
respectively. The geometric structure can be described 
as a frustum of a cone (truncated cone). The diameter of 
the manipulator is 40mm at the base and narrows down 
to 10mm at the tip. The robot is composed of three main 
components: an inner air-tight and stretchable latex 
bladder, an outer, non- stretchable (but foldable) 
polyester fabric sleeve and 12 nylon tendons. The 
stretchable cylindrical latex bladder is inserted into the 
cylindrical polyester sleeve. The outer sleeve is 30cm in 
length when fully inflated. As the fabric material is non- 
stretchable, the outer sleeve prevents any ballooning of 
the inner bladder in radial direction beyond its 
maximum diameter. Whilst morphing from a deflated to 
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an inflated state, the robot can only expand along its 
longitudinal axis (elongation). The stiffness of the arm 
can be controlled adjusting the tendons - e.g., tightening 
the tendons at a fixed air volume in the sleeve will 
reduce the robot's length and increase the stiffness. The 
nylon tendons are guided along the outside of the 
manipulator sleeve within polyester channels, 90° 
spaced apart along the perimeter of the outer sleeve 
(Figure 2(c)). In our three-section prototype, four 
tendons are fixed to the top of each section of the 
manipulator (Figure 2 (a)). As Figure 2 shows, the soft 
manipulator is mounted inside a phantom dome. The 12 
tendons that are connected to the three sections are 
controlled by a system of 12 pulleys. Each pulley is 
fixed onto the shaft of a stepper motor (Soyo 
SY57ST56-0606B Unipolar Stepper Motor). By 
controlling the angular position of the pulley system, it 
is possible to vary the length of each tendon separately 
and, hence, the bending of the manipulator. 



Fig. 2 (a) Conceptual system architecture of the manipulator 
with three sections and a combination of tendon-driven and 
pneumatic actuation. CAD drawings of the soft manipulator: 
(b) 3D and (c) section view. 

The stepper motors are connected to bi-polar micro- 
stepping drivers (Sparkfun Big Easy Driver ROB- 
11876). The inner latex bladder is connected to one 
pressure regulator (SMC ITV0010-3BS-Q). The 
regulator is able to control the air pressure from 
O.OOIMPa to O.IMPa. An in-built pressure sensor 
ensures to maintain the desired pressure inside the 
bladder. The motor drivers and pressure regulator are 
interfaced via three DAQ cards (NI USB -6211) to 
LabVIEW software. A joystick (Logic 3 JS282 PC 
Joystick) is utilized as an input device for the operator. 

EXPERIMENTS AND RESULTS 

A series of experiments has been conducted in order to 
explore the stiffness of manipulator. Lateral forces F 1? 
F 2 , F 3 have been exerted to the manipulator, see Figure 
3. The experimental results were recorded and are 
plotted (Figure 3). The internal pressure of the bladder 
was altered from 0.015 MPa to 0.055 MPa and, hence, 
the stiffness increased. The maximum force held by the 


robot was defined as a displacement at the tip of 5 mm 
was reached. The manipulator is able to hold up to 
5.2 N at the tip. 



Fig. 3 Experimental results of lateral forces F h F 2 and F 3 . 


DISCUSSION AND FUTURE WORKS 

We have presented here our new hybrid and 
antagonistic actuation system for a continuum soft 
robotic manipulator fusing pneumatic with tendon- 
driven actuation. Being inspired by the biological role 
model, the octopus, our antagonistic actuation system 
aims at modelling the octopus way of using its 
longitudinal and transversal muscles in its arms: 
activating both types of muscles, the octopus can 
achieve stiffening of its arms - this concept is realised 
here using tendons and pneumatics to configure the arm 
and to adjust its stiffness, as shown by the experimental 
study. In future works, we will explore ways to utilise 
the empty space within the air-filled bladder, to 
integrate sensors and actuators for additional tools as 
may be required in the surgical context. 
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INTRODUCTION 

Intrauterine interventions currently rely on the use of a 
fetoscope as the main intraoperative imaging 
modality [1]. Fetal surgery could benefit from 
computational processing of fetoscopic images, such as 
image mosaicking. However, images from fibre-optic 
fetoscopes contain a “honeycomb” pattern due the 
camera oversampling the irregular fibre bundle image 
guide. This honeycomb pattern distracts the clinician 
and will adversely affect any subsequent image 
computing algorithm; thus, a crucial first step in the 
image-processing pipeline is the removal of the 
honeycomb pattern. 

Previously published methods for the removal of the 
honeycomb structure include adaptive spectral 
masking [2] and spatial interpolation methods [3-5]. We 
chose the spectral masking filter approach because, as 
opposed to spatial interpolation, it can cope with 
defocused images where individual fibres cannot readily 
be distinguished in the honeycomb pattern, and because, 
once the method is validated, spatial filtering can be 
readily optimised by GPU implementation. To filter out 
the honeycomb structure without over-blurring the 
image, it is important to estimate the optimal cut-off 
frequency f c for the filter mask. The optimal f c depends 
on the fibre intercore distance and also on the camera 
focus and zoom levels, which may change during the 
procedure. We test three different approaches for 
estimating the optimal f c from fetoscopic images: (1) by 
direct measurement of the intercore distance; (2) by 
estimating the fibre density; or (3) by estimating the 
honeycomb frequency spectrum. Here we demonstrate 
that the honeycomb pattern can be robustly filtered 
using the f c estimated from any of the three approaches. 

MATERIALS AND METHODS 

Data. Test images were obtained using a fibre-optic 
fetoscope and an imaging system (Karl Storz, 
Germany). The test images include a set of in vivo 
images obtained during an intrauterine procedure. 

Estimating f c from the intercore fibre distance. The 
locations of the fibres were identified by finding the 


regional maxima in the image. Then the median 
distance between adjacent fibres d ic was used to 
determine f c as: f c = 0A/d ic . 

Estimating f c from the fibre density. The imaged fibre 
density was estimated by measuring the area of the 
imaged bundle and using the number of fibres in the 
bundle, n fibre , supplied by the manufacturer. Then the 
average distance between adjacent fibres d ic was 
calculated by relying on a hexagonal circle packing 

assumption as: d ic = J(2 * area)/(rif ibre * V3). As in the 
first approach, f c was then set as: f c = 0A/d ic . 

Estimating f c in the frequency domain. The FFT of the 
fetoscopic image was computed and its log amplitude 
was used to estimate the frequency of the honeycomb 
pattern. The honeycomb frequency f comb was detected by 
integrating the log amplitude of the FFT over rings of 
varying radius, and finding the non-zero radius with the 
largest average amplitude (Fig. 1). f c was then 
determined as: f c = 0.4 * f comb . 

Spectral mask filtering. Using the f c estimated with 
either spatial or frequency method, fetoscopic images 
were filtered by applying a smoothed spectral mask in 
the frequency domain. A circle mask of radius f c was 
smoothed with a Gaussian filter to avoid ringing 
artefacts (std. dev. = 10) and multiplied with the FFT of 
the image (Fig. Id). The filtered image was then 
recovered by taking the inverse FFT of the masked 
spectrum, and then examined for the removal of the 
honeycomb pattern. 

RESULTS 

The optimal f c was estimated using all three approaches, 
and the resulting values were within 15.68+2.96% 
(mean + std. dev., n=28 images) of one another. The 
estimates from the first and the third approach differed 
by 11.04+2.51%. No one method was systematically 
overestimated or underestimated the f c . 

Images of the USAF-1951 test pattern at different zoom 
and focus levels were filtered with the spectral mask 
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b) 



Fig. 1 a) Magnified fetoscopic image of the USAF-1951 test 
pattern showing the honeycomb pattern, b) Fourier power 
spectrum of the whole fetoscopic image (logarithmic grey 
scale), c) Filtered image using estimated f c = 0.068, d) Ring 
used for the detection of f comb (inset) and the smoothed mask 
used for spectral filtering. 

using the f c estimated in the frequency domain, which 
was generally the median value of the three estimates. 

All filtered images had the honeycomb pattern removed 
while not over-blurring the small structures in the image 
(Fig. 1). 

In vivo images filtered using the estimated f c also had 
their honeycomb patterns removed while preserving the 
finer anatomical details in the image (Fig. 2). 



Fig. 2 a) In vivo fetoscopic image (dashed rectangle: ROI), b) 
Magnified ROI showing the honeycomb pattern, c) Filtered 
fetoscopic image using estimated f c = 0.064, d) Magnified 
ROI of the filtered image. 


a) b) c) 


Fig. 3 a) Filtered image using a Gaussian filter with o = 
0.4*d ic , b) The same as in a) but with c = 0.5*d iC; c) Filtered 
image using the estimated f c as in Fig. lc. 


Finally, a comparison between the adaptive spectral 
filtering and the Gaussian filtering shows that the 
Gaussian filtering slightly over-blurs the image, while 
the spectral filtering maintains the edges (Fig. 3). 

DISCUSSION 

Our results suggest that the honeycomb pattern can be 
robustly removed from fetoscopic images at different 
focus and zoom levels by estimating the optimal cut-off 
frequency and applying a filter. The cut-off frequency 
for the filter mask could be estimated using any of the 
three proposed approaches. Future work will focus on 
speed optimisation by re-estimating f c only when a 
change in zoom or focus is detected, and designing a 
fast filter as done in [6] but with spectral characteristics 
that are closer to an ideal circular band-pass filter 
without the ringing artefacts. 
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INTRODUCTION 

Minimally invasive surgery, both robotic and 
laparoscopic, has demonstrable benefits for the patient 
[1]. However, inability to distinguish different tissues 
and target lesions reduces eligibity for laparoscopic 
surgery. Image guidance using previously acquired 
imaging has the potential to mitigate these effects 
increasing eligibity for minimally invasive surgery. We 
present the "SmartLiver" image guidance system for 
image guided laparoscopic and robotic liver resection. 

Existing image guidance systems for the liver [2,3] rely 
on separate probes to locate points on the liver surface 
relative to the theatre coordinate system. With 
sufficiently long digitizing probes these systems can be 
used for laparoscopic as well as open surgery. However, 
as well as requiring an additional tool in theatre they do 
not locate the laparoscope relative to the liver, 
precluding the use of augmented reality on the 
laparoscope's display. The SmartLiver guidance system 
addresses these two short comings by using a tracked 
stereo laparoscope to locate the liver surface. 



Fig. 1 Reconstructed Patches up to 40 cm 2 are composited 
using tracking data to enable rigid ICP registration. 


Registration of the composited patches to the segmented 
liver surface is performed using a rigid iterative closest 
point algorithm. Initialisation of the registration is 
performed manually. An initial estimate of the liver 
position is refined by using a second object tracked by 
the Spectra to move the liver in six degrees of freedom. 
Any tracked object can be used for this. We use an off 
the shelf sterile object so the alignment can be 
performed by the surgeon. 


This paper describes the system, results of validation 
experiments on porcine and phantom models and initial 
results from first use in humans. 

MATERIALS AND METHODS 

Prior to surgery, contrast enhanced CT images of the 
patient are segmented 1 to surface models of the relevant 
anatomy. During surgery the HD stereo laparoscope 
(Viking HD) acquires image pairs of the liver surface. 
Fast feature matching [4] and triangulation are used to 
reconstruct surface patches relative to the laparoscope. 

In practice the size of these patches is limited to less 
than 40 cm 2 by the narrow stereo separation (4.5 mm) of 
the stereo laparoscope. A patch of this size is 
insufficient to enable registration of the liver surface, so 
multiple patches are composited using the position of 
the laparoscope determined by an Optotrak Polaris 
Spectra tracking system. Figure 1 shows 9 such patches 
overlaid on a silicone phantom used for development 
and validation. 


Segmentation by http://www.visiblepatient.com/ 


The software is implemented using the NifTK[5] 
platform. The system has been tested and the accuracy 
assessed on a silicone phantom and 3 pigs. Accuracy 
was measured by picking landmarks on the video and 
model, and measuring the distance between the 
landmarks on a plane parallel to the video screen, see 
Figure 2. 



On screen error in pixels 


Fig. 2 System errors were calculated by manually picking 
landmark points in the video, and measuring their distance to 
the model points on a plane parallel to the screen. 
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RESULTS 

On the phantom the system achieved RMS errors of 2.9 
mm based on 791 samples of 9 subsurface landmarks. 
The 3 porcine experiments gave an average RMS error 
of 9.4 mm based on between 234 and 483 samples of 4 
to 6 surface landmarks. 


The results of the pig and phantom experiments were 
analysed to determine the contribution of different error 
sources to the overall system error. Understanding the 
error sources enables the construction of a system 
development road map, Figure 3, showing how ongoing 
developments should impact the system performance. 


Registration was achieved in under 5 minutes on the 
phantom, and in under 15 minutes on the pigs. The 
longer time taken on the pigs was due to the increased 
difficultly in performing initial manual registration 
under the influence of non-rigid and dynamic 
deformation, mainly due to breathing motion. 

The system has been used on four human cases to date 
with more ongoing. The human cases have 
demonstrated that the system can be used practically 
within current clinical practice. There appears to be less 
dynamic deformation (breathing) in the human cases 
and the liver anatomy is significantly simpler, indicating 
that the accuracy on human cases could be better than in 
the pigs. However, we are currently having to adapt the 
system to cope with the smaller human liver and less 
available landmarks for accuracy assessment. 

DISCUSSION 

The SmartLiver system presented here enables image 
guidance during laparoscopic liver resection. The 
current system has been tested in-vivo on pigs and on 
humans and shown to enable registration and 
visualisation in theatre. The system accuracy is 
currently around 9 mm on pigs, but is likely to be 
slightly better on humans. 

Human trials are ongoing with the current system. We 
are measuring the clinical outcomes together with Likert 
Scale based questionnaires to assess the system 
performance during the human trials. By correlating 
these measures with assessment of the system accuracy 
and user interface design, we hope to show how these 
parameters could affect clinical outcomes. 
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Fig. 3 The results can be used to isolate sources of error and 
create a development road map for this and other abdominal 
image guidance systems. 


Using the NifTK software platform we have developed 
several pluggable modules to improve the system 
performance. We have implemented a deformable 
registration algorithm though at present the surface 
patches do provide sufficient constraints for this to 
work. We are also investigating using subsurface 
landmarks imaged using laparoscopic ultrasound. 
Development of these novel algorithms is ongoing in 
parallel to the clinical implementation of the basic 
system. 
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INTRODUCTION 


Magnetic resonance imaging (MRI) provides high 
resolution soft and hard tissue imaging where its 4 non- 
ionizing radiation property is ideal for paediatric 
patients. However, despite these advantages, it remains 
primarily a diagnostic device because of the challenges 
in developing non-ferrous tools that can work in the 
magnetic field without impacting image quality and tool 
performance. MRI-compatiable robots have been 
developed in the areas of neurosurgery, prostate and 
breast biopsy but they are not capable of imaging and 
treating bone conditions. [1-4] 

Bone tumours account for 6% of paediatric patients 
where they are most common between ages 10-20. [5] A 
bone biopsy involves taking a core sample (~4.4mm 
diameter) of the bone with a manual hollow serrated 
needle. Clinicians have used MRI to target the lesion 
and avoid injuring critical vessels and nerves but it 
requires moving the patient in/out of the MRI scanner 
and using an adhoc needle placement. The biopsy 
procedure is complex as it requires a significant amount 
of force and coordination to manually obtain the core 
sample in the MR environment. 

To address this problem, we have developed a 
modular MRI-guided paediatric surgical robot (PSR) for 
bone biopsy. The system is designed to allow the 
clinician to simultaneously image and perform the 
procedure with greater precision and less time. 

MATERIALS AND METHODS 

The PSR is developed as a collaborative effort 
between interventional radiology, neurosurgery and 
private sector (Engineering Services Inc). The major 
system requirements for PSR included: 3T MRI 
compatibility, in-bore intra-operative capability, 
mechanical rigidity and strength sufficient for bone 
drilling. The system diagram is shown in Figure 1. 



Fig. 1 - PSR System Diagram. 


The goal is to allow the clinician to select the bone 
target from pre-operative imaging and then allow the 
PSR to take the core sample while simultaneously 
imaging to track the needle location. 

The system is designed to be modular and 
reconfigurable to accommodate for a variety of pediatric 
patient sizes and potential applications. There are 5 
generic modules: linear motion module, turret module, 
elbow roll module, wrist tilt module and penetration 
module. The module casings were created in plastic 
using stereo lithography (SLA) and ultrasonic motors. 
Non-ferrous metal brass was used as fasteners and 
gearboxes. The linear module provides a translation 
stroke motion of up to 30mm with a mass of 0.3 kg. The 
turret and elbow roll module are rotational modules that 
provide up to 360 degree rotation with a mass of 0.25 
kg. The wrist tilt module is design a “horseshoe” where 
it secures and rotates a tool. The penetration module 
provides linear translation for up to 90mm and it has a 
flexible force sensor (FlexiForce A201) mounted. 

For the bone biopsy procedure, a tool module is a 
designed to mimic the manual bone biopsy procedure. 
It carries a clinical serrated hollow titanium biopsy 
needle (In vivo) with three concentric cylinders: a 
pointed solid rod for initial anchoring, hollow serrated 
cutting drill bit and a solid protective sleeve. This 
allows the system to take 25 mm bone core sample. See 
Figure 2 - Right. 

A mobile arch system has been developed that 
translates the PSR along the axis of the MR table. The 
modules can be secured in a variety of orientations such 
as on the MRI table or arch to reach different clinical 
targets. The PSR has up to 5 degrees of freedom with an 
additional separate tool actuation. See Figure 2 - Left. 



Fig. 2 - Left: PSR System, Right: Surgical Tool Module. 

The robot controller box is designed to be placed 
within the MR room beyond the 5 Gauss field line. It is 
interfaced to the control PC through a penetration panel. 

For MRI compatibility testing, the system was 
mounted on the MR bed in a Philips Achieva 3.0T MRI 
scanner and two phantoms (Philips mineral oil phantom 
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and a watermelon) were used for imaging testing. The 
phantoms were scanned with a Flex-M coil. A series of 
interventional Tl-w, T2-w and T2*-w imaging 
sequences were conducted in both axial and coronal 
planes (2D fast spin echo (FSE), 2D fast gradient echo 
(GRE), 2D FIESTA and 3D fast spoiled gradient 
(FSPGR)). Each of the sequences were conducted under 
3 states (system - power on, system - power on, system 
- motors in motion). 

For drilling performance, PSR drilling tests were 
conducted on polyurethane foam blocks (density 15-40) 
that simulated human cancellous bone and chicken 
bone. 


fresh chicken bone samples. See Figure 4. For the bone 
testing, core samples were successfully retrieved from 
flat regions of the bone. 



Fig. 4 - Left: Bone phantom, Right: Chicken Bone. 


RESULTS 

The workspace of the system is described in table 1 
(joint limits). The system is capable of translation 
linearly along the MR table bed up to 100mm and the 
surgical biopsy needle can be detached and sterilized 
separately from the robotic system. The system can be 
driven in joint and Cartesian space. 


Table 1 - PSR Joint Workspace 


Vertical linear module 

30 mm 

Turret roll module 

140 degrees 

Elbow roll module 

140 degrees 

Wrist tilt module 

40 degrees 

Arch linear motion 

100 mm 

Drill penetration module 

90 mm 

Biopsy tool module 

25 mm 


MR imaging testing showed that the PSR did not 
adversely impact the MR scan sequences (Figure 3) 
with minimal change to the image quality and signal to 
noise ratio for the all the imaging sequences. This was 
for PSR present (no power), PSR present (powered on) 
and PSR moving. 



A) power off B) power on, w/o motion C) power on and motions 



A} power off B) power on, w/o motion C) power on and motions 



A) power off B) power on, w/o motion C) power on and motions 

Fig. 3 - Top: FSE, Middle: FGRE, Bottom: FIESTA. 

Results from drilling testing showed that the PSR is 
capable generating sufficient force to drill through 
simulated bone phantoms with 15-40 density as well as 


DISCUSSION 

During drilling, it was noted that if the PSR was 
attempted to drill a narrow cylindrical sample (diameter 
of bone < drill), the serrated needle would not be able 
to “catch” the bone and slip off the surface. This 
problem also appeared during off-centre insertion where 
the PSR drill bit was not completely on the surface. In 
these scenarios, it appeared the overall system stiffness 
of the PSR was not sufficient to hold the system in place 
during an off-centre or high curveature drilling. 
Additional work is being conducted to analyze and 
improve the overall system focusing on gearbox 
backlash and the plastic frame. In addition, the drilling 
algorithm is being refined with clinician input to mimic 
the circular and axial insertion motion to account of off- 
centre bone sampling. 

To improve the range of motion, a motorized 
rotational arch mechanism is being designed that will 
allow the PSR to rotate around the MRI bore. From an 
integration prospective, pre- and intra-operative 
software is being developed to allow the clinician to 
image, plan and execute the procedure from a dedicated 
console using the forward and inverse kinematics. 

The PSR demonstrated that it can be operated within 
the MR bore during imaging with minimal impact to 
image quality. The system is able to able drill and take 
core samples from simulated bone samples and chicken 
bone while simultaneously imaging the target. Further 
invivo animal testing is being planned to validation its’ 
performance. 
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INTRODUCTION 

Direct manipulation of a robot is a common 
approach in human robot cooperative settings, 
furthermore it is ideal for human intervention in safety 
critical situations. For direct manipulation of compliant 
robotic arms, two critical issues must be resolved: 
weight compensation and compliant control. Weight 
compensation minimises the payload on the operator’s 
hand by compensating the mass of tools and robot. 
Compliant control accurately senses the operator’s 
manipulation intention and moves the arm accordingly. 
A number of design strategies have been proposed to 
address these two issues. The Baxter robot (Rethink 
Robotics, USA) has elastic joints with springs for 
weight compensation. However, these non-rigid joints 
have problems of low accuracy and low stiffness over 
the arm. When using rigid joints on robotic arms, in the 
Lightweight Robot (LWR) (Kuka Roboter, Germany) 
and the MIRO (DLR, Germany), multiple single-axis 
torque sensors are fixed on all joints to sense the torque 
raised from the distal end-effector side or the other part 
of the arm body. Precise weight calibration of the tool 
and the robot is required to conduct the weight 
compensation and to prevent motion drift when the 
operator leaves the robot arm with the tool alone. 
Changing tools may incur system instability due to the 
weight difference or moment of inertia difference 
among different tools. It is practically inconvenient to 
re-calibrate the system for each time changing the tool. 

In this paper, we propose a calibration-free 
cooperative manipulation method. In brief, a force 
sensor is mounted between the tool and a manipulation 
knob held by the operator. Tools with different weight 
can be freely exchanged during the manipulation 
without tool recalibration or change of settings. The 
sensitivity range for the manipulation force can be 
adjusted in a wide range according to the specific 
requirements. From the experimental results with a tool 
insertion task, this method has a similar outcome 
compared to the joint level torque sensing method 
implemented on the LWR. 

MATERIALS AND METHODS 

The proposed calibration-free cooperative 
manipulation method is designed to be suitable for most 
robotic manipulators since it only relies on the robot’s 
position controller. A force sensor is installed to the 
end-effector to measure the hand manipulation force. 
Unlike conventional methods that have the sensor 
mounted between the robot and the tool, the force 
sensor is placed such that the manipulation force is 


decoupled from the tool’s weight. An example setup is 
shown in Fig. 1. The control scheme for cooperative 
manipulation is implemented by commanding the 
robotic end-effector position according to the force 
measurements of the sensor. First, the sensor’s raw 
reading is subtracted with the initial force to account for 
force offset readings and the weight of the knob. To 
generate motion from the manipulation force, the force 
reading F is converted to velocity by multiplying it with 
a constant gain value K v . This gain can be adjusted 
depending on the desired sensitivity. In order to reduce 
the noise during an idle state, a threshold F t is applied 
beforehand. This threshold also provides stability by 
imposing an initial force required to move the tool. 

V v = K v (F-F t ) (1) 

F _ ( F, F <Th 
1 [Th, F > Th {) 

where Th is the initial force parameter. 

V v is then fed as an input for a trajectory generator to 
produce a position command for the robot. A maximum 
acceleration parameter, is used to limit the acceleration 
of the robot at this stage. 

An experiment is setup to evaluate the performance 
of the proposed method as well as to compare it with the 
joint torque sensor based approach. The setup is 
illustrated in Fig. 2, which consists of a light-weight 
Hamlyn Arm and a Mini40 Force/Torque Sensor by 
ATI Industrial Automation (NC, USA). The Hamlyn 
Arm is a 6 degrees-of-freedom robotic arm 
incorporating harmonic drive gears with a new magnetic 
absolute angular position encoder. It is also equipped 
with a motion controller that performs PID position 
control on-board. 

An asynchronously running inverse kinematic 
solver, using the damped-least- square (DLS) Jacobian 
based algorithm [3], converts the desired pose in task 
space to a position command in joint space. 

The control software is implemented in C++ running 
on Scientific Linux 6.6 with a real-time kernel. 
Reflexxes Velocity-based Type II On-Line Trajectory 
generator [2] is used in task- space to convert a velocity 
command to a position command. The same setup can 
also be applied to the LWR. 

RESULTS 

A tool insertion task is used to validate the proposed 
methods on the LWR. The task for each participant is to 
pick-up and move the tool tip from a starting point P 0 to 
the entry point of the tube Pj by holding only the knob 
with one hand. The tool should be guided along the 
centreline of the tube until the tooltip reaches the exit 
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point P 2 . Participants are asked to perform the task three 
times, with different tool weights and control scheme 
between tasks. 



Fig. 1 System Overview: (A) Arm, (B) Force Sensor, (C) 
Tools for the insertion task, (D) Variable tool weight 


The light and heavy tool are weighted 0.8kg and 
1.4kg respectively. The control methods used are the 
LWR’s gravity compensation and our force sensor 
based control method. Therefore, there are four 
variations of parameters for this experiment in total. 
They are denoted as LWR-Light, LWR-Heavy, FT -Light, 
and FT -Heavy. 



Fig. 2 Hamlyn Arm, with enlarged view of its joint 

In order to evaluate the effects of manipulating an 
un-calibrated tool, the LWR is only calibrated with the 
light tool. Hence, the user is expected to perceive 0.6 
kg-equivalent tool weight with LWR-Heavy 
manipulation task. The orientation of the arm is always 
maintained in the horizontal pose by setting its 
rotational stiffness to the maximum. In case of the 
proposed methods, the LWR runs in position control 
mode. 


Table I Experimental results showing the mean values ± SD 
and minimum-maximum values in parenthesis. 



LWR-Light 

LWR-Heavy 

FT-Light 

FT-Heavy 

Time (s) 

4.33 + 1.94 
(2.67-9.47) 

5.30 + 3.06 
(2.25-12.54) 

4.91 + 1.51 
(3.17-7.85) 

5.08 + 2.19 
(2.99- 9.48) 

Deviation 

(mm) 

3.89 + 1.84 
(0.14-9.27) 

3.83 + 1.90 
(0.06-9.63) 

3.47 + 1.70 
(0.04-10.14) 

3.22 + 1.53 
(0.06-7.43) 

Force (N) 

7.32 + 0.91 
(3.88-11.66) 

11.98 + 1.53 
(6.20-15.28) 

1.44 + 0.43 
(0.0-4.81) 

1.42 + 0.35 
(0.01-3.72) 


Each participant is given 10 minutes to get familiar 
with the system in all task variations. There are total of 
5 participants who took part in the experiments, 
resulting in 60 recorded tasks. The results are 
summarised in Table I. Only the manipulation within 
the tube region (between P : and P 2 ) is analysed while 
the rest is discarded. Three performance metrics for 
each are calculated: completion time, position deviation 
from the tube centreline, and manipulation force. A 
demonstration video of the control scheme implemented 
on the Hamlyn Arm and LWR is available online at 
http ://goo. gl/MU9nB 3 . 

DISCUSSION 

In this paper, we presented a calibration-free gravity 
compensation method based on tool mounted force 
sensor. It has a wide range of manipulation sensitivity 
settings to account for manipulation tasks where some 
friction is required for motion steadiness. In tasks where 
small manipulation forces are preferred, a sensitivity as 
low as 0.03 N is achieved. 

Implementing gravity compensation with joint 
torque sensors, requires complex joint construction as 
well as complex low level-joint torque control 
algorithms. The proposed methods achieve similar 
functionality with comparable performance for simple 
manipulation tasks. Additionally, the manipulation force 
can be adjusted down to a lower level. All these can be 
implemented on any robotic arm with a position 
controller, which is simple in terms of hardware and 
software. 

One disadvantage of our method is that contact 
forces between the tool and the environment are not 
detectable. In applications where the contact force must 
be limited, direct positioning of a force sensor at the 
tool tip or active constraints can be implemented to 
overcome this issue. 
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BACKGROUND 

The last two decades have witnessed a paradigm shift to 
an endovascular, minimally invasive approach in 
treating diseases of the aorta and its branches, with clear 
advantages in terms of morbidity and mortality [1,2]. 
All endovascular procedures are currently performed 
under fluoroscopic X-ray guidance; radiation exposure 
is therefore important to consider. Technically complex 
cases inevitably take longer to perform, increasing 
radiation exposure. Complex patient anatomy and 
suboptimal fluoroscopic imaging can make 
endovascular manipulation challenging. Conventional 
endovascular catheter technology contributes to these 
technical impediments; with a limited repertoire of 
shapes, they are difficult to steer and lack active 
manoeuvrability. This can hinder overall stability and 
control at key target areas requiring frequent catheter 
changes, risking loss of access. Relatively inaccessible 
anatomical targets may lead to prolonged 
instrumentation carrying significant risks to the patient 
(embolization, stroke, vessel wall trauma) with 
increased fluoroscopic exposure and long overall 
procedure times for both patients and staff. 
Endovascular robotic technology in the arterial tree is a 
novel concept, however pre-clinical outcomes are 
encouraging and clinical experience is growing [3,4,5]. 

METHODS 

Radiation exposure data was collected prospectively 
during 19 consecutive FEVAR cases over a 14-month 
period at a tertiary referral centre. 10 of these cases 
involved robotic manipulation of the renal arteries. 
Robotic manipulation is a new technology and a novel 
concept in endovascular intervention, entailing an 
inherent learning curve. Previous work has shown this 
learning curve to be shorter than conventional manual 
techniques [6] . Operators in this study were experienced 
in both conventional (n>300) and robotic (n>30) 
approaches. 

Digital over-lead dosimeters measured operator 
exposure (OE) during pre-defined procedural stages. 
Patient exposure (PE) for the entire procedure was also 
measured. Cannulation technique (robotic/manual), 
operator positioning, c-arm angulation, procedure and 
fluoroscopy times were noted. Non-parametric tests 


were used for comparisons. The robotic system is 
remotely-operated, however it should be noted that 
“negligible” readings were recorded as the dosimeter 
sensitivity threshold (0.03mSv cases 1-10; O.OlmSv 
cases 11-19) rather than 0. 

RESULTS 

Median OE per case was 0.37mSv[IQR(0.23-0.65)] 
compared with 1.68mSv (0.98-2.84) for PE. OE 
readings per stage were: stent alignment [0.03(0.03- 
0.27)], renal cannulation [0.03(0.03-0.03)], visceral 
cannulation [0.08(0.03-0.22)], completion [0.11(0.03- 
0.26)]. OE during visceral cannulation was significantly 
higher compared to renal cannulation (p<0.001). 

There were no significant differences in procedure time 
and OE between manual and robotic techniques, despite 
the robot being used for more anatomically challenging 
cannulations. OE was significantly affected by operator 
positioning, with the highest doses observed in the 
cephalad position [0.09(0.03-0.27)] during visceral 
vessel manipulation via the axillary/brachial approach 

(p<0.001). 

CONCLUSIONS 

Operator and patient radiation exposure for FEVAR is 
considerable and higher than reported figures for 
conventional EVAR. With increasing caseloads and 
anatomical complexity, a significant cumulative OE is 
likely. Radiation awareness and staff education is 
therefore essential. Alternative approaches such as 
remote robotic navigation for high-risk stages should be 
explored. 

FEVAR = Fenestrated Endovascular Aneurysm Repair 
EVAR = Endovascular Aneurysm Repair 
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[5] Bonatti J, Vetrovec G, Riga C, Wazni O, Stadler P. 
Robotic technology in cardiovascular medicine. Nat Rev 
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Fig. 1 Classification of operator positioning. 
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Fig. 2 Occupational exposure by position: OE was 
significantly affected by positioning (p=0.005), with the 
highest being at “Cephalad” and “Patient Right A” positions. 
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INTRODUCTION 

Nowadays robotic prostatectomy is widely performed 
for minimally invasive prostatic tumor resection. It uses 
stereoscopic laparoscopes to intuitively visualize the 
organ surface in the body and manipulate various 
surgical tools. Unfortunately, the insufficient light 
sources and the relatively narrow field of view of the 
binocular stereoscopic laparoscope usually limit the 
high-quality visualization on the organ being operated 
on and its anatomical surroundings. Moreover, these 
limitations affect surfuce structures (e.g., neurovascular 
bundle) and disable surgeons to accurately perceive the 
surgical field (e.g., subtle bleeding areas), potentially 
increasing risks in robotic prostatectomy. 

This study aims to accurately recognize subtle surface 
structures and visualize regions of interest in prostate 
surgery. Based on color constancy [1], the human visual 
system has a relatively constant awareness of the color 
of objects under illumination differences. This implies 
that the human color perception generally depends on 
the reflectance of objects but not the scene illumination. 
Hence, the fidelity and quality of images or videos can 
be augmented by removing the illumination component. 

This work is basically motivated by color constancy. 
To model human color constancy, Retinex is currently a 
popular and effective approach [2] . This paper proposes 
Retinex processing to deal with the aforementioned 
limitations occurred in robotic prostatectomy. Retinex- 
augmented visualization enables surgeons to accurately 
identify subtle surface details and plainly perceive the 
anatomical surroundings of the organ being operated on. 

MATERIALS AND METHODS 

The Retinex-augmented visualization approach consists 
of three main steps: (1) Gaussian convolution, (2) color 
restoration, and (3) histogram equalization. Fig. 1 shows 
the flowchart of Retinex processing for augmentation. 

Gaussian convolution. Suppose that (x, y) is a pixel 
on a color image I c with three channels of red, green, 
and blue c E {R,G,B}. For each pixel / c (x,y) in each 
channel, we perform the Gaussian convolution and the 
logarithmic transformation and obtain 0 c ( x >y) : 

< P c (x,y ) = log I c (x,y) - log (G m (x,y) * I c {x,y ) , (1) 
Gm(x,y) = K m ex p[- (x 2 + y 2 )/2a £], (2) 

where m indicates the Gaussian scale level, o m denotes 
the scale, and K m guarantees / G m (x,y)dxdy = 1. 

Gaussian convolution and logarithmic transformation 
in Eq. (1) aim at accurately separating the reflectance 
and illumination components on image I c . Based on the 




Gaussian convolution 


Histogram equalizer 


Augmented output 


Color restoration 


Fig.l Retinex processing for augmented visualization. 


definition of Retinex model, the output R c (x,y) is the 
sum of weighted 0 c (x,y) in different Gaussian levels 

M 

R c ( x > y) = ^ oj m 4>c( x ’ y), (3) 

771 = 1 

where a) m is the weight in various levels and the level 
number M. Typically, we set M = 3 and co m = 1/M. 

Color restoration. Since the processing of Gaussian 
convolution and logarithmic transformation deteriorates 
the color saturation and generates a grayish image, a 
color restoration step is necessarily performed to obtain 
the color restoration coefficient y c (x, y) 


Yc(.x,y) = a 


log£/ c O,y) - log 


I 


I c (x,y) 


> (4) 


ce{R,G,B } J 

where a is a gain factor and /? controls the nonlinearity. 
After that, the output becomes to be y c (x, y)R c (x, y). 

Histogram equalization. Although the restoration 
step corrects the color saturation, it usually results in a 
problem of pixel inversion, i.e., the pixel intensity close 
to 0 in color channels can jump to 255 while the pixel 
intensity near 255 also potentially change to 0. The 
histogram equalization is used to tackle this problem. 

Blind quality assessment. As it is somewhat difficult 
to generate ground truth for clinical video visualization, 
a blind video quality measure is used to quantitatively 
evaluate augmented video visualization. The Shannon 
entropy is widely employed to blindly assess the quality 
of images without any reference, and is given by [3] 

EOc ) = -Z ce{R , G , B }Zf=i//ai)log//(7i)< (5) 

which calculates the image entropy by characterizing its 
histogram (L t is the frequency of i-th intensity level). 
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Fig.2 Examples of augmented visualization by Retinex processing: Original (left) and augmented (right) images. 

—B— Original — e — Augmented —a— Original — e — Augmented 



Fig-3 Information entropy comparison of original and augmented videos recorded from robotic prostatectomy. 


Validation. The Retinex augmentation was evaluated 
on clinical videos that were collected from the robotic 
prostate surgery using a da Vinci surgical Si system. 

RESULTS 

Fig. 2 illustrates surgical video images with and without 
the Retinex augmentation. The image visual quality was 
obviously improved and more subtle structure details 
were visible after the Retinex augmentation. Fig. 3 
quantitatively compares the entropies of original and 
augmented images and proves Retinex effective. 

DISCUSSION 

This work showed the first study on augmented surgical 
filed visualization for releasing subtle surface structures 
and anatomical surroundings of the prostate. In robotic 
prostatectomy, the organ sub -structures or surroundings 
are hidden and hardly perceived by surgeons due to 
limited illumination and narrow endoscopic view of the 
surgical field. Based on the color constancy theory, the 
Retinex augmentation strategy was proposed to address 
the limitations. The Retinex processing uses Gaussian 


convolution and logarithmic transformation to separate 
the reflectance and illumination components. By 
removing the illumination component, the visual fidelity 
was augmented. The experimental results prove that this 
study is effective to enhance the surgical vision, 
improving a surgeon’s awareness of various structures. 
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INTRODUCTION 


MATERIALS AND METHODS 


High quality surgical education has long been in the 
focus of medical associations throughout the world. 
Between 2000 and 2002, the Accreditation Council for 
Graduate Medical Education of the United States 
defined six fields of competency for physicians that 
were later implemented in the surgical field by the 
American College of Surgeons (ACS). These 
competencies included basic knowledge, clinical 
competence, interpersonal skills, professionalism, 
patient-based learning, system based learning and 
surgical simulation [1, 2]. In the past years, computer 
assisted surgery (CAS) has also become part of the 
everyday surgical practice, leading to a dramatic 
decrease in the time required for carrying out the actual 
surgical procedure, especially when the medical 
professionals receive training in robotic surgery [3]. 
However, most of the commercially available surgical 
robots do not provide features for surgical training, 
leaving university hospitals to solve this issue by 
alternatives, e.g., using retired robots as educational 
platforms. 

Camera handling, along with many other laparoscopic 
procedures, lacks standardization background in general 
surgical practice and training. While today’s training in 
this field is mostly based on learning-along-the-way 
methods, untrained medical students may compromise 
the success of surgery and reduce the effectiveness of 
the procedure in terms of communication and dynamics. 
There are numerous robotic solutions existing for 
camera handling, including the ones equipped with a 
separate hand panel [4], driven by voice control [5] or 
navigated by the movements of the surgeon’s head [6], 
although up until now there is no general ‘best’ solution 
for the issue of separate handling of the camera(s) in 
laparoscopic surgery. The aim of this research is to 
develop a complete system for education of robotic 
minimally invasive surgeries, and surgical skills 
assessment. The primary focus is the camera handling 
during laparoscopic interventions, where a novel design 
of a camera holding robotic arm is presented that may 
be integrated into the da Vinci Surgical System of 
Intuitive Surgical Inc. 


The da Vinci Surgical System has undoubtedly become 
the most successful complete surgical system available 
in commercial distribution. As of December 31, 2014, 
there were 3,266 units installed worldwide, and the 
manufacturing company is employing almost 3,000 
people in total [7]. In order to enhance research in 
surgical robotics, the da Vinci Research Kit (dVRK) 
was developed at Johns Hopkins University (and partner 
institutions) that allows researchers to access all levels 
of control of the da Vinci system or its modules [8]. The 
retired da Vinci Surgical System at Obuda University, 
Budapest does not provide endoscopic camera handling, 
which motivated this research to develop a novel 
system, the Computer Assisted Laparoscopy (CALap) 
for overcoming this issue, leading to the design of a 
low-cost robotic arm used for holding laparoscopic 
cameras. The used training environment was developed 
at the Semmelweis University, Budapest in 2008, called 
the Apollo box trainer [9] . The environment follows the 
classical layout of laparoscopic box trainers, including a 
non-opaque cover, four input ports in order to 
manipulate the trocars and an aperture for camera 
installments. 



Fig. 1 The assembled Computer Assisted Laparoscopy robot 
(CALap) at the Antal Bejczy Center for Intelligent Robotics, 
Obuda University. 
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COMPUTER ASSISTED LAPAROSCOPY ROBOT 

Due to the lack of functional camera handling at the 
dVRK and da Vinci systems available at our laboratory, 
a new low-cost assembly was created that would be 
used in education and skill assessment. The prototype of 
the CALap system is shown in Fig. 1. The first 
prototype was created in a short period of time. The 
hardware was designed as follows. Aluminum profiles 
were used to hold the main weight of 1 kilogram, which 
provide the necessary stability of the construction 
without significantly increasing the total weight of the 
system. Special shaped parts for gears and bearing 
holders were also custom designed and 3D printed. The 
electronic implementation of the CALap system is 
based on the master-slave design, which allows low- 
level electronic and high level programming approach. 
Low-cost lightweight stepper motors were used for 
driving the assembly, keeping it as lightweight as 
possible. Communication between the master side 
computer and the slave robot was achieved by a custom- 
designed serial protocol, transferred over USB 
connection. 

The programming of the system constitutes of two 
phases: the dVRK programming and creating the 
software for the CALap system. The da Vinci boards 
use the IEEE- 1394 FireWire communication lines for 
data transfer. The protocol allows a high bandwidth for 
the dVRK, resulting in a near real-time communication 
speed. The created software on the dVRK side is built 
up from 5 layers: the hardware interface, the low-level 
control, the high-level control, teleoperation and 
application. The CALap software was utilizing the 
master-slave concept. The custom-designed serial 
communication protocol incorporates driving 
commands from the master side computer to the slave 
robot, while position and status data are received and 
transmitted back to the program. The software at the 
slave side runs on an Arduino Uno board using an 
ATmega328 microcontroller, attached to the CALap 
robotic arm, interpreting serial communication, 
executing low level motor driving commands and 
reporting back to the master computer. The full 
experimental setup and environment of the CALap 
system is shown in Fig. 2. 

DISCUSSION 

Computer assisted and robotic laparoscopic surgical 
procedures are relatively new, rapidly developing fields 
of general surgery. The new approaches to surgical 
interventions require surgeons with psychomotor 
adaptations, which are not part of the classical surgical 
curriculum. Due to the new requirements in the field, 
there is an increasing need of revolutionizing surgical 
education, including special training methods and skill 
assessment. The introduced laparoscopic camera 
handling system is a useful tool for the above mentioned 
topics, while its lightweight and low-cost (~$100) 
design allows one to apply it in many different types of 
surgical interventions and environments. 



Fig. 2 The experimental setup of the CALap system with the 
da Vinci Surgical System and Research Kit in the background. 

In the future, the CALap system would be a great tool 
for providing robotic background for surgical training in 
the field of minimally invasive surgery. Further plans 
include complete integration with the dVRK system. 
This prototype would also open up opportunities for 
bringing low-cost and lightweight robotic tools into the 
operating room. 
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INTRODUCTION 

Needle steering robots — robotic systems that insert 
flexible needles along controlled curved paths through 
tissue — have been discussed extensively in the research 
literature. Although significant work on modeling, 
control, path planning, and image guidance has been 
completed, no practical needle steering system has been 
evaluated for efficacy in a specific clinical application. 
The goal of our current work is to apply robotic needle 
steering to radiofrequency ablation (RFA) of liver 
tumors, by steering electrodes along curved paths to 
previously inaccessible regions of the organ. 

Multiple needle steering techniques have been 
described, including needle-base manipulation, tissue 
manipulation, active cannulas, programmable bevels, 
and bevel-tip or bent-tip needles [1]. While these 
methods have shown promising initial results, they have 
been validated almost exclusively in artificial tissue 
simulants. Based on the limited results reported, the 
same needle steering techniques perform worse (i.e., 
they result in less tightly curved paths) in ex vivo and in 
vivo biological tissues [2]. 

As part of our ongoing work, we are currently 
developing methods to improve the curvature of bent-tip 
steerable needles in liver tissue. This necessitates a 
specific design requirement. In other words: How much 
curvature is enough? The goal of the work described in 
this paper was to establish the curvature requirements 
for steerable needles in percutaneous RFA of liver 
tumors, by combining analysis of medical images with 
knowledge of existing clinical techniques and needle 
steering models. 

MATERIALS AND METHODS 
Medical Image Analysis 

An open-source contrast-enhanced abdominal CT 
scan of a healthy 36-year-old male was analyzed [3]. 
The segmented liver capsule was exported from 3D 
SLICER, refined using MESHLAB, and imported into 
MATLAB to serve as a workspace boundary. Similarly, 
large-diameter portions of the internal vasculature 
which represent a significant bleeding risk (specifically 
the hepatic and portal veins with diameter over 
approximately 5 mm) were imported to serve as 
obstacles. These models are shown in Fig. 1. 

Two entry vectors v ^ were defined based on typical 
introducer placement for percutaneous RFA of liver 
tumors. Vector v ^ represents an intercostal approach 


into the right liver. Vector v ^ represents an anterior 
subcostal approach under the costal margin into the left 
liver. Both orientations reflect insertion under 
ultrasound guidance with the needle at 45° to the 
transducer axis. 

Reachable Workspace 

For each v^ l \ the corresponding reachable 
workspace R ^ was measured by discretizing the liver 
volume L (excluding the vasculature) into 4-mm voxels, 
and determining if each voxel could be reached by a 
permissible path. Paths that crossed the liver boundary 
(capsule) or vasculature were excluded to decrease risk 
of bleeding and other complications. Paths were also 
restricted to have one or two constant-curvature 
sections. Although insertion and rotation of a bent-tip 
steerable needle can theoretically generate more 
complex needle paths, in practice the mechanical 
properties of liver cause these complicated paths to 
relax into simpler paths. Demonstrating this, Fig. 2(a) 
shows ultrasound segmentation results from switching a 
steerable needle on a straight path to a maximally 
curved path in ex vivo porcine liver. After the direction 
change, rather than forming a path with distinct tangent 
sections as predicted, the needle relaxed by 10.3 mm to 
a constant curvature path. Two-section paths were 
excluded if the waypoint was already beyond the target 
(i.e., if the dot product of the vector from the waypoint 
to the target and the tangent vector at the waypoint was 
negative) to avoid impractical looping behavior. Finally, 
paths were excluded if they required radius of curvature 
p below threshold p m in , which was varied as a parameter 
from 10-200 mm. Fig. 2(b) shows examples of 
permissible and excluded paths. 

RESULTS 

Fig. 3 shows the size of the reachable set — specified 
by the ratio of the set sizes |R^|/|L| — as a function of 
minimum radius of curvature p m in . With /v n =100 mm, 
paths starting at v ^ were able to reach approximately 
8% of L, while paths starting at v ^ were able to reach 
approximately 41% of L. 

Fig. 4 shows a frontal slice near the center of the 
liver, with color indicating radius of curvature p 
required to reach each voxel from The needle was 
not able to reach areas behind the hepatic veins and 
surrounding the insertion. Most of the slice was 
reachable using single-section paths, except for the area 
blocked by the vessels. 
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Fig. 1 Models of the liver boundary (capsule), large hepatic 
veins (green) and large portal veins (blue). The entry vectors 
are also indicated. The superior-inferior (SI), medial-lateral 
(ML), and anterior-posterior (AP) directions are shown. 
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Fig. 2 Needle paths: (a) Ultrasound segmentation results 
showing a steerable needle before and after a direction change 
in ex vivo porcine liver. Rather than forming a path with 
distinct arc sections, the needle relaxed into a single arc. (b) 
Reachable and unreachable voxels within L , and lines 
showing, from left to right, an excluded path that violates /v n , 
an excluded path that passes through an obstacle, and a 
permissible two-part path. 

DISCUSSION 

Results shown in Fig. 3 reveal the gap between 
current needle steering methods and our clinical 
application. The tightest steerable needle curvatures p m in 
reported in biological tissue are between 100-200 mm 
[2]. Even combining the intercostal and anterior 
subcostal approaches, such needles would only permit 
access to 50% of the liver at best. Since the distribution 
of tumor sites is approximately uniform throughout the 
liver, current steerable needles would be unable to 
access many targets. Although angling the introducer 
could change the results shown in Fig. 4, interference 
from the ribs and potential for tissue injury limit this 
degree of freedom after the initial insertion. 

An important potential advantage of needle steering 
in RFA of liver tumors is the ability to reach targets in 
the dome of the liver (on the right-lateral superior 
surface), which can be very difficult in some patients 


using traditional RFA probes due to interference from 
the ribs. 
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Fig. 3 Ratio of reachable set size to total liver size as a 
function of minimum radius of curvature p^. Results for 
intercostal ( v (O) and anterior approaches are shown. 
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Fig. 4 Frontal slice of L . Colors indicate the radius of 
curvature p required to reach each voxel. Black voxels were 
not reachable with any curvature in the examined range (10- 
200 mm). White voxels, including the vessels inside the liver, 
were outside L. 

As seen in Fig. 4, a radius of curvature of 
approximately 50 mm is necessary to reach the right- 
lateral aspect of the liver dome. We believe p m in = 50 
mm is thus a reasonable curvature requirement for 
future development, since it allows the needle to access 
the liver dome, and reach approximately 85% of the 
liver volume from the entry points we have described. 
Improvement of needle curvature to this level will 
require significant refinement of needle steering 
techniques. 

This work was supported in part by the National 
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INTRODUCTION 

With robotic-assisted surgery becoming more common 
practice in urology, effective training remains a 
challenge. There is a considerable learning curve 
associated with robotic training, though this has been 
reported as gentler than for laparoscopic training 1 . 
Simulation has gained wide acceptance as a method of 
reducing the initial phase of the learning curve. The 
RobotiX Mentor™ is a new virtual reality simulator 
which mimics the interface of the da Vinci® Surgical 
System, with integrated training modules including 
Fundamentals of Robotic Surgery (FRS) and Robotic 
Suturing. Face and content validity of the suturing 
module have been confirmed in a previous study 2 , 
though this was performed on a different platform. This 
study aims to assess face, content and construct validity 
of the RobotiX Mentor™. It also aims to assess its 
acceptability as a training tool and feasibility of its use 
in training. 


MATERIALS AND METHODS 

This prospective, observational and comparative study 
recruited novice (n=20), intermediate (n=15), and expert 
(n=ll) robotic surgeons as participants from institutions 
across the United Kingdom and at the 30th European 
Association of Urology Annual Meeting. Each 
participant completed nine surgical tasks across two 
modules on the simulator, followed by a questionnaire 
to evaluate subjective realism (face validity), task 
importance (content validity), feasibility, and 
acceptability. Outcome measures of novice, 
intermediate, and expert groups were compared using 
Mann-Whitney U-tests to assess construct validity. 
Outcome measures of novice, intermediate, and expert 
groups were compared using Mann-Whitney U - tests to 
assess construct validity. 


RESULTS 

Construct validity was demonstrated in a total of 17/25 
performance evaluation metrics (p<0.001). Experts 
performed better than intermediates in regard to time 
taken to complete the first (p=0.002) and second 
(p=0.043) module, number of instrument collisions 
(p=0.040), path length (p=0.049), number of cuts >2mm 
deep (p=0.033), average distance from suture target 
(p=0.015), and number of suture breakages (p=0.038). 
Participants determined both the simulator console and 
psychomotor tasks as highly realistic (mean: 3.7/5) and 
very important for surgical training (4.5/5), with system 
pedals (4.2/5) and knot tying task (4.6/5) scoring highest 
respectively. The simulator was also rated as an 
acceptable (4.3/5) tool for training and its use highly 
feasible (4.3/5). 


CONCLUSIONS 

Construct, face, and content validity were established 
for the RobotiX Mentor and feasibility and acceptability 
of incorporation into surgical training was ascertained. 
The RobotiX Mentor shows potential as a valuable tool 
for training and assessment of trainees in robotic skills 
and may reduce the initial learning curve if utilised as 
an adjunct to operating-room training. Investigation of 
concurrent and predictive validity is necessary to 
complete validation and evaluation of learning curves 
would provide insight into its value for training. 


REFERENCES 

[1] Goldstraw MA, Challacombe BJ, Patil K, Amoroso P, 
Dasgupta P, Kirby RS. Overcoming the challenges of 
robot-assisted radical prostatectomy. Prostate cancer and 
prostatic diseases. Mar 2012;15(l):l-7. 

[2] Abboudi H, Khan MS, Aboumarzouk O, et al. Current 
status of validation for robotic surgery simulators-a 
systematic review. BJU international. Feb 
20 1 3 ; 1 1 1 (2) : 1 94-205 . 


69 



5 n 



Console Training Modules 


Figure 1. Assessment of face validity by novice, intermediate, and expert groups. Each element was scored on a numerical scale 
between 1 (not realistic) and 5 (very realistic). The diathermy pedals and ‘4 th Arm Cutting’ task were rated as most realistic, whereas 
graphics and ‘Knot Tying’ task were judged as least realistic. 
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Figure 2. Assessment of content validity by intermediate and expert groups. Each task was scored on a numerical scale between 1 
(not important) and 5 (very important). ‘Knot Tying’ and ‘Vertical Defect Suturing’ tasks were rated as most important, whilst 
‘Puzzle Piece Dissection’ was judged as least important. 


70 


Re-Thinking Patient Specific Instrumentation and Robotics in 
Orthopaedics: A New Mechatronic Approach 

A. Darwood 1 , R. Emery 2 , R. Richards 3 , P. Reilly 2 , 

F. Rodriguez y Baena 2 , A. Dawood 1 , A. Tambe 4 

Prometheus Surgical Ltd., London, UK, alastairdarwood@hotmail.com 

2 Imperial College, London, UK, r.emery@imperial.ac.uk 
3 Cavendish Implants, London, UK, 4 Derby Hospital NHS Trust, UK 


INTRODUCTION 

It is well known that the accurate placement of 
orthopaedic hardware such as arthroplasty prostheses 
into a biomechanically derived optimum position is of 
paramount importance in the long term success of these 
interventions 1,2,3 . Similarly, in many orthopaedic 
procedures it is advantageous or even essential to know 
the exact placement of hardware such as screws or pins, 
for example in spinal surgery or complex fracture 
reduction. Two main technologies have been developed 
to facilitate accurate placement: bespoke patient specific 
instrumentation such as rapid manufactured guides and 
orthopaedic surgical robots. Both technologies enable 
good results but are hindered by well-known significant 
drawbacks such as high cost, complex logistical chains, 
poor versatility and long set up times 4,5 . 

This paper presents a radically new technology 
developed to overcome many of the drawbacks of 
existing approaches whilst providing a means to 
accurately place orthopaedic hardware according to a 
pre-operative plan. The approach allows accurate 
bespoke drill/cutting guides to be rapidly produced 
intraoperatively at minimal per patient cost using a 
simple low cost hardware element and disposable 
consumables. In this paper, the technology is described 
in the context of total shoulder arthroplasty, more 
specifically, the placement of the glenoid component 
according to a pre-operative plan. 

MATERIALS AND METHODS 

The method comprises two main intraoperative 
components: A mechatronic apparatus and disposable 
sterile consumables. The apparatus is made up of an 
optical 3D scanner (HDI 120, LMI Technologies, Delta, 
BC Canada), a 2 axis Computer Numerical Control 
(CNC) drill with replaceable sterile drill bit and a 3 axis 
CNC receptor assembly for the sterile disposables. The 
sterile disposables comprise a plastic element with 
standard geometry coated on one side with a body of 
mouldable material able to be rapidly solidified, for 
example by exposure to air, heat or light and a carrier 
element to allow an interface with the non-sterile 
surfaces of the receptor assembly. The disposable 
component is inserted into the receptor assembly in a 
known way, by virtue of its standard geometry. With the 
exception of the mouldable element, all elements of 
both components remain spatially and positionaly 
calibrated with respect to one another. 



Fig. 1 : Schematic and internal photograph of robot showing 
surface scanner (1) and CNC apparatus (Drill and gimbal 2) 
with disposable components (3). 

The approach consists of 6 steps to produce an 
intraoperative guide. 

1. Planning is carried out on CT or MRI patient data. 
This process defines e.g. a guide wire position as a 
vector within the coordinate frame of reference of the 
digital scapula model. This plan may be carried out pre- 
operatively or intra-operatively. 

2. The surgeon exposes the surgical site in the 
conventional manner. In this case, exposing the osseous 
anatomy of the glenoid cavity. 

3. The sterile disposable consumable is prepared. The 
mouldable side of the component is pressed into the 
surgical site (the glenoid cavity) and quickly solidifies 
forming a mould of the glenoid cavity and exposed 
portions of its rim. The component is removed and 
placed within the sterile carrier which in turn is placed 
into the corresponding receptor assembly of the 
apparatus. 

4 . The optical scanner scans the surface of the mould, 
containg the detail of the glenoid cavity geometry. The 
scan data is inputted into a software registration module 
based on the classical Iterative Closest Point (ICO) 
algorithm, which registers the glenoid surface mould 
scan with its corresponding position on the CT/MRI 
imaging derived digital patient anatomy. In this way, the 
scanned glenoid surface model is automatically placed 
into the same coordinate frame of reference as the 
original 3D patient scapula data. As a result, the vector 
describing the location of the guide wire may now 
intersect the digital glenoid surface mould model. 

5. From this data, a CNC drilling path is now created. 
The CNC drill fitted with a sterile drill bit and the CNC 
receptor assembly is repositioned according to the 
output of the planning module, which is based on the 
registered scan data, in order to drill through the mould 
and carrier such that the hole drilled matches the 
geometry of the guide wire vector passing through the 
digital glenoid surface mould model. 

6 . The sterile moulded part is removed from the 
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Fig. 2: Internal robot photograph - Following registration the 
sterile drill (red arrow) has cut a guide hole through the 
mouldable component (Green arrow). It may now be removed 
from the robot ready for use. 

machine and returned to the operative site. Its solidified 
moulded geometry constrains it to the identical positon 
from where the mould was taken. The prepared guide 
hole may now be drilled through by the surgeon placing 
a guide wire into the pre-planned position. Once this is 
complete, the disposable may be slid off the guide wire 
and discarded, leaving the wire in the pre-operatively 
planned position. Surgery may then continue in the 
conventional manner. 


Fig. 3: The modified disposable element ‘locks’ into place by 
virtue of the solidified impression on its underside. A guide 
wire (indicated by red arrow) can now be drilled through the 
guide hole. 

A prototype device was constructed from a combination 
of ‘off the shelf components and specifically designed 
3D printed parts. Bespoke software was created to 
control the above steps to be carried out in one 
computer program. The prototype is controlled either 
with the use of a built in touch screen PC or an external 
desktop computer. The prototype is able to demonstrate 
a full operative surgical workflow including the 
maintenance of sterility of all parts that come into 
contact with the patient and operated by scrubbed 
personnel. A dental mould material was used for 
prototype testing however it is conceived that any 
sterile, non-toxic rapidly setting mouldable material 
may be used in its place. Full ex- vivo testing was 
carried out to assess the feasibility of the approach in 
terms of accuracy, manufacture time and ease of use. 

10 anonymized shoulder CT scans of patients requiring 
arthroplasty were collected and thresholded to produce a 
3D model of patient anatomy. A consultant orthopaedic 
surgeon with arthroplasty experience planned the ideal 
guide wire position in each scapula in order to place the 
glenoid component into an ideal position. Three planar 
markers were also added to each glenoid model so as to 
create a reference plane upon which to compare guide 
wire placement accuracy. A pair of each scapula glenoid 
was 3D printed (Objet Eden, Stratasys company USA) 
to produce two groups of 10 identical models: a guided 
group and a conventional group. The surgeon manually 


placed a guide wire into all glenoids in the conventional 
group according to the bespoke plan created for each 
glenoid. The prototype was then used in the guided 
group to place guide wires according to identical 
bespoke plans for each glenoid. Guide wire placement 
was subsequently analysed in both groups and 
compared to the pre-operative plan. Accuracy was 
measured in two ways: minimum distance between 
planned glenoid entry coordinate and achieved vector 
axis of physical guide wire (mm) and absolute angle (°) 
between planned guide wire axis and achieved physical 
guide wire axis. 

RESULTS 

Full guide manufacture was completed in approximately 
5 minutes including planning, molding and device set 
up. In the conventional group, average placement 
accuracy was 1.580mm (3.22-0.4mm) and 6.815° 
(16.63-1.19°) in comparison to the guided group 
achieving 0.47mm (1.3 1-0. 12mm) and 1.572° (1.77- 

0. 06.). Student’s t-tests were applied to the minimum 
distance and the absolute angle metrics and they were 
both shown to be statistically significantly affected by 
the method (manual vs. automatic) at the 95 % 
confidence interval (p < 0.05). 

DISCUSSION 

With a total guide manufacture time of 5 minutes, cheap 
consumables, a simple hardware unit and the ability to 
accurately place orthopaedic hardware into a pre- 
planned position it is evident that this approach 
overcomes many of the drawbacks of existing 
orthopaedic guidance technologies. The approach may 
be used in the guidance of other arthroplasty operations 
with minimal re-configuration of the disposable 
components. Work in these areas is ongoing at time of 
writing. In conclusion, this paper describes a new 
approach and validation for the intraoperative rapid 
production of bespoke patient specific cutting guides. In 
removing many of the drawbacks of existing 
technologies and significantly reducing costs, it is hoped 
that the technique may open up orthopaedic guidance so 
that a greater number of patients may benefit from a 
fully computer guided orthopaedic intervention. 
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INTRODUCTION 

Robotic devices in medicine have yielded numerous 
benefits from advanced diagnostic tools to instruments 
that augment surgeons' physical capabilities [1,2]. With 
smaller and safer technologies, medical technology is 
advancing from interventional robotic medicine to 
personalized robotic medicine with the aim of reducing 
clinical assistance, hospitalization, or the repetitive need 
for surgeries. Robotic implants, as a new category of 
medical robots, are implanted in the body to 
autonomously regulate biological processes for long 
periods of time. An example of their clear benefit is 
long-gap Esophageal Atresia (LGEA) in which a section 
of the esophagus longer than 3cm is missing [3,4]. 
Long-gap EA afflicts over 100 babies per year in the 
USA and Europe. One effective procedure to repair this 
congenital defect is Foker’s technique [5,6], in which 
growth-inducing traction forces are applied to the 
esophageal segments using suture that is tied off outside 
the child's back. The traction is manually maintained for 
an average of 14 days to induce sufficient elongation to 
close the gap. While producing superior outcomes to 
alternative procedures, the child is sedated and kept on a 
ventilator for the duration of the treatment. It would be 
extremely beneficial to these patients if (1) multi-week 
sedation could be avoided so as to eliminate any effect 
on long-term neurocognitive development, (2) the need 
for re-operation due to suture tear-out could be reduced, 
and (3) the duration and cost of hospitalization could be 
reduced. 

We present a robotic implant that has the potential to 
resolve these shortcomings. The implant operates in the 
thoracic cavity attached to the two esophageal segments 
using attachment rings, thus localizing the elongation 
procedure. In contrast to the standard Foker technique, 
the implant’s configuration is not affected by infant 
motion and, consequently, the child does not need to 
remain paralyzed and sedated in the ICU during 
traction. Equipped with force and position sensors, the 
motorized implant allows controlled and precise tissue 
traction, eliminates the risks of suture tear-out due to the 
ring attachment that distributes forces around the 
esophagus, and enables traction monitoring. The design 
concept of the robot was introduced in [7]. 

In this paper, we present the robotic implant, which has 
been designed and tested for in vivo animal 
experiments. The following are the paper’s 
contributions: (1) a robotic implant design featuring a 
biocompatible encapsulation, (2) an integrated system 


for remote implant monitoring and control, and (3) 
position and force control of esophageal tissue. 



Fig. 1 Robotic implant. A. Implant and monitoring system. 
B. Schematic for testing in swine. C. Implant shown attached 
to length of porcine esophagus. Note: initial tests induce 
growth of esophageal segment between rings. 


IMPLANT DESIGN FOR SURVIVAL TESTS 

The entire system, shown in Fig. 1A, consists of the 
robotic implant, a control box, battery pack, and a 
laptop. An animal protocol, approved by IACUC 
(Institutional Animal Care and Use Committee), 
provides for two types of experiments. In this first, the 
robotic implant is mounted on an intact esophagus and 
growth is induced in the segment between the rings over 
a 7 -day period (Fig. 1 BC). In subsequent experiments, 
a section of esophagus is removed to mimic actual 
atresia and the rings move to close the gap. 

The robotic implant applies traction on the esophageal 
segments using a motor, which displaces the rings 
relative to each other. The implant is also equipped with 
force and position sensors to measure and control tissue 
forces and displacements [7]. The implant is sealed 
with a biocompatible encapsulation. We designed the 
implant encapsulation with the following properties: (1) 
biocompatible; (2) anti-fouling; (3) impermeable to air 
and water; and (4) abrasion resistant. It is fabricated 
from silastic (Fig. 1C), which satisfies the first three 
properties. The silastic (Bentec Medical) embeds a 
polyester mesh to satisfy the forth requirement. Silastic 
caps were fabricated using a molding process (Dow 
Corning Corp.). Thorough sealing was ensured using 
medical adhesive silicone Type A (Dow Corning 
Corp.). The tubing protecting the electric wires was also 
made of silastic (Bentec Medical). 
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The attachment rings were fabricated from stainless 
steel by TIG welding to preserve the material properties 
and avoid corrosion. The rings are detachable and 
adjustable in order to allow easy implantation and 
suturing, depending on the diameter of the esophagus. 
The control box contains an Atmel controller (Baby 
Orangutan B-328, Pololu), force sensor amplifier, and 
Bluetooth module (BlueSMiRF, Sparkfun Electronics). 
The control box is worn in a vest pocket by the animal 
during the survival experiments. The laptop is used to 
launch a graphical interface for visualizing real-time 
sensor data, inputting controller commands and data 
logging via Bluetooth. A low-level controller generates 
traction forces using either force or position set-point 
control. Set points can be set manually or using a high- 
level controller to produce a desired waveform, e.g., 
adjustment once per day or every few minutes. 
Automated adjustment during animal studies enables 
controlled comparison of traction histories and their 
effect on tissue growth rate and properties. 

IMPLANT CONTROL 

The controllers were tested in 36-hour ex vivo studies 
with swine esophagus during which the implant and 
esophageal segment were submerged in saline. The 
saline kept the tissue wet while also testing the 
impermeability of the encapsulation. The esophageal 
segments were taken from swine weighing 43 kg on 
average. The initial relaxed length of the esophagus 
segment was 20 mm (Fig. 1C). The laptop was used to 
command the implant and to log position and force data 
during testing. There were no liquid leaks inside the 
robotic implant during and after these experiments, as 
observed from the signals of the electronic components 
and from visual inspection of the interior of the robot. 
Figure 2 shows implant signals corresponding to 
automatic position control. Position and force signals 
are reported for a 1200s duration during which the 
commanded displacement was a square wave of 
amplitude 10mm and a period of 130s. 
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Fig. 2. Robot position set-point control. Response to commanded 
10mm position square wave. 


Under position control, the implant generated a peak 
force in the tissue of about 2N, which reduced to about 
1.5N after one minute due to relaxation of the tissue. 



Fig. 3. Robot force set-point control. Response to force- 
control set points of 0.8N and 1.0N. 


Figure 3 presents the response to force set-point 
commands of 0.8N and 1.0N obtained during long-term 
testing. The plots also show the response after the 
controller was turned off at 0.1138h. Force fluctuations 
occur due to relaxation of the tissue. 

IN VIVO TESTING 

The implant is sterilized using the Ethylene Oxide (EtO) 
sterilization process in order to protect the electronic 
components. It is implanted through a right 
thoracotomy, which provides access to the esophagus 
while keeping a safe distance from the heart. The cable 
from the implant is tunneled to the control box in the 
vest through the skin. The animal is fed a semi-solid 
diet for the duration of the experiment. Traction is 
applied every day. The tissue growth is assessed by 
comparing the stretched segment with a control 
segment, and by histological analysis. 
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INTRODUCTION 

The widespread growth of robotic minimally invasive 
surgery has led to the need for a standardized training 
curriculum. Toward that end, the Institute for Surgical 
Excellence is conducting the Fundamentals of Robotic 
Surgery (FRS) validation trial. This multi-site study is 
testing both novice and experienced surgeons using 
Intuitive Surgical da Vinci robots and robot simulators. 
While a standardized curriculum will increase training 
consistency, the tasks performed on the robot itself will 
still rely heavily on expert observation of a trainee’s 
performance. While common in traditional surgical skill 
assessment, observation-based ratings are both 
subjective and time consuming for the expert reviewers. 

Thus, there is also a growing need for more objective 
and automatic approaches to surgical skill assessment 
during robot-based training [1]. Although several 
studies have assessed surgeon performance by analyzing 
the robot arm motions, e.g., [2], these analyses do not 
account for the quality of the physical interaction 
between the instruments and their environment. Our 
work on VerroTouch, a system that provides real-time 
haptic feedback of instrument vibrations [3], has shown 
that instrument vibrations are a construct-valid measure 
of surgical skill during robotic in vitro training tasks 
such as suturing and peg transfer [4]. While instrument 
vibrations can capture rough interactions between the 
instruments and with stiff tissue, some contacts with 
soft tissue do not produce measurable instrument 
vibrations [5]. We hypothesize that the quality of these 
interactions can be monitored by examining the forces 
that the robotic instruments exert on the tissue; we can 
measure such forces during in vitro and ex vivo training 
tasks but not during in vivo surgery. 

To better understand how instrument vibrations and 
contact forces may elucidate the skill level of a surgeon 
during various tasks, we have recorded surgeon 
interactions with the ex vivo avian tissue model used 
during the robot-based portions of the FRS validation 
trial at our institution. This paper provides a descriptive 
analysis of the vibrations and forces caused by a single 
expert robotic surgeon in this study. 

MATERIALS AND METHODS 

One experienced colorectal surgeon (>50 robotic cases 
on human patients) performed a series of psychomotor 
tasks with a da Vinci Si robot as part of the FRS 
validation trial taking place at the University of 
Pennsylvania. The five tasks included knot tying, 


suturing, fourth-arm cutting, pattern cutting, and vessel 
energy dissection. The tasks were performed on an 
avian tissue model (a turkey leg quarter) as a post-test to 
the FRS training curriculum. 

The da Vinci Si was augmented with our system for 
measuring instrument and camera vibrations, as well as 
the forces applied to the tissue (Fig. 1). A three-axis 
accelerometer (LIS344ALH) measured the vibrations of 
the left tool, the right tool, and the camera; each sensor 
was attached using a plastic bracket on the shaft. A 
force sensor (ATI Mini40) was mounted under the plate 
holding the tissue model; the three vibration axes were 
combined into a single magnitude, and the force 
measurement was separated into normal and tangential 
components. We also recorded the camera video feed. 

We measured the vibrations and forces during three 
selected FRS tasks. Their instructions were as follows: 

A. Suturing: Close the incision using a continuous 
horizontal mattress suture pattern, with the needle 
entering and exiting through the marks placed on 
the skin. The suture is already anchored on one side 



Fig. 1 A da Vinci Si robot augmented with accelerometers that 
measure instrument and camera vibrations. The avian tissue 
model is resting on top of a draped plate that is equipped with 
a three-axis force sensor. 
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Fig. 2 Vibration and force recordings for the (A) suturing, (B) pattern cutting, and (C) vessel energy dissection tasks performed on 
the avian tissue model. Shaded grey regions highlight key events that are visible in the vibration and/or force data. 


of the incision. At the end of the closure, tie the 
suture to itself using a surgeon’s knot - a double 
throw, followed by two single throws. Tools: (2) 
Large Needle Drivers. 

B. Pattern Cutting: Cut and peel the skin off exactly 
along the marked line, avoiding damage to the 
underlying muscle tissue. Tools: (1) Maryland 
Bipolar Forceps, (1) Curved Scissors. 

C. Vessel Energy Dissection: Dissect down to the 
turkey femoral artery or vein. Liberate a (>3 cm) 
segment of the vessel from the surrounding tissue, 
cauterize at two points (>0.5 cm apart), and cut the 
vessel in between those points. Tools: (1) Maryland 
Bipolar Forceps, (1) Curved Scissors. 

To provide an analysis of time-synced events in the 
force and vibration data, the experimenter considered 
the time-series signals alongside the recorded video. 

RESULTS 

The Suturing task data shows large vibration spikes on 
both tools when the needle is passed from hand to hand 
(see Fig. 2A). The normal and tangential force exerted 
on the tissue increases when the surgeon pulls up on the 
tissue to drive the needle or pulls tight on the suture to 
close the incision. A negative normal force can also 
occur when the surgeon pushes on the tissue with one 
tool while pulling on the suture with the other. 

The Pattern Cutting task generates more vibrations than 
Suturing, especially from the scissors (see Fig. 2B). 
With sufficient counter-tension, the vibration caused by 
the cut is also transmitted to the Maryland forceps. A 
few large vibration spikes are caused by collisions 
between the instruments. Both the tangential and normal 
forces exhibit peaks when the surgeon pulls on the 
tissue to create counter-tension for the cut. The negative 
normal forces typically stem from the surgeon pushing 
on the tissue to separate the skin from the fat. 

The Vessel Energy Dissection task shows the strongest 
vibrations of the studied tasks (see Fig. 2C); these 
vibrations occur when the surgeon bluntly dissects the 
tissue, cuts tissue during the dissection, and cuts the 
vessel near the end of the task. Large vibrations also 


result from instrument collisions. There are both 
positive and negative normal forces exerted on the 
tissue, with corresponding tangential forces; these 
forces are the result of the surgeon pressing and pulling 
on the tissue to locate and dissect out the vessel. 

DISCUSSION 

This paper described the instrument vibrations and 
contact forces generated by one expert surgeon 
performing three FRS tasks on an avian tissue model. 
These recordings are consistent with those from other 
surgeons in the FRS trial. However, with further 
analysis, we do expect to find variation between 
subjects in different surgical specialties, especially 
when novice surgeons are considered. These results will 
hopefully unravel the unique signatures within the 
vibration and force data that can be used to not only 
identify a given task, but also automatically evaluate 
how skilled a surgeon is at performing that task. 
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INTRODUCTION 

A few endoscopic procedures are performed with a 
fluid-immersed endoscope. Fetoscopy is one such 
minimally invasive procedure that allows observation 
and intervention within the amniotic sac during 
pregnancy. The feto scope is inserted through the uterus 
and immersed in amniotic fluid. Fluid has a strong 
influence on the image formation process due to 
refraction at the interface of the fetoscopic lens. It is 
determined by the optical properties of the amniotic 
medium (Fig. 1). 


Fig. 1 Images of a calibration target acquired with a fetoscope. 
Left: calibration setup. Right: images of the calibration target 
respectively in the air and in water. 

Accurate calibration is critical to vision-based methods 
for providing image-guided surgery and real-time 
information from the surgical site [1]. It consists of 
recording images of a calibration target of known 
geometric pattern in order to estimate optical properties 
of a camera. In the case of a fetoscope, calibration 
should be realised in the amniotic fluid which currently 
cannot be practically achieved. A fluid -immersed 
calibration is also impractical for sterilisation purposes. 
Few computer vision methods address this issue and 
most of them are not adapted to the wide field of view 
as well as the severe distortion effect of a fetoscope [2]. 
We experimentally show a direct link between dry and 
fluid-immersed camera parameters that can compensate 
for the optical properties of amniotic fluid as well as 
radial distortion effects. 

ENDOSCOPIC CAMERA MODEL 

Endoscopic cameras generally embed large field of 
view optical systems. We chose to rely on the unifying 
model of central catadioptric system proposed in [3] to 


handle severe distortion effects and assume optical 
parameters remain fixed. This model unifies under a 
common mathematical formalism central catadioptric 
optical systems with a single projection centre. It has 
also been proved that the unified model can suit 
perspective as well as certain fish-eye optical systems 
which makes it flexible. 

The image formation model, illustrated in figure 2, 
consists of four mappings. A projective ray joins a 3D 
point Q of the scene with the effective projection center 
O intersecting the unit sphere in a single point Qs. The 
second mapping is equivalent to projecting the point Qs 
into the normalized plane from a novel projection center 
O’=(0,0,-QT. This non-linear mapping intrinsically 
handles severe radial and tangential distortions. 
However, a fourth degree function based on a Brown- 
Conrady model was considered to compensate for lens 
aberration and model approximations. The last 
transformation maps the distorted point qd to the point q 
observed in the image plane. It is expressed by the 
internal camera parameter matrix K which embeds the 
parameter y (directly related to the focal length / of the 
camera), a skew parameter (pixel ratio) and the position 
of the principal point in the image plane. It is worth to 
note that y is directly related to c according to the real 
shape of the omnidirectional camera considered [4] . 



Fig. 2 Unifying model of central catadioptric system. 

We now consider a thin interface, planar and fronto 
parallel to the image plane, which separates the optical 
system of the endoscope from the external medium (as 
it appears to be in practice). Because it is difficult to 
accurately calibrate the depth of the interface we 




77 



arbitrarily fixed it above the normalized plane. 
According to Snell’s law the refraction angle of the 
incoming light ray is expressed by: 

? i ± sind 1 = n 2 sin# 2 (1) 

Where ri\, n 2 represent the refractive index of each 
medium at either end of the interface and 0i, 0 2 represent 
the angles between the incoming ray and the normal ft 
to the interface. We approximate the refractive index of 
the amniotic fluid by the refractive index of water 
^1^1.33. According to the unified model, we expect to 
estimate a new value and a new focal length y’ to 
compensate for the refractive properties of fluid. The 
new £’ is obtained thanks to: 

f = arg min f + (q uy - (?) 

where q u corresponds to the coordinates of a 2D point 
on the normalized plane and Q’ is an immersed 3D 
point casting on the normalized sphere on point Qs. 
According to Snell’s law (1), Q’ lies on the light ray 

belonging to the plane defined by (JQ V Q s ) and n and 
forming an angle 0 2 with n at Qi. The unifying model of 
central catadioptric system allows to derive the value of 
y’ according to £’ and the shape of the catadioptric 
mirror. This relationship cannot be applied to a 
fetoscope which does not embed a real catadioptric 
system. However, our experiments, presented in the last 
section, show that an appropriate value of y’ can 
compensate for the refraction effects without the need to 
estimate other camera parameters. 

METHOD 

The calibration method can be decomposed in the 
following steps. We first realize a dry calibration of the 
fetoscope using [4]. We use one of the calibration image 
to estimate the light paths inside the optical system from 
each pixel of the image plane to the 3D points Qi 
belonging to the interface plane. We then use equation 

(1) to compute new incoming light paths considering the 
refractive index of the external fluid. We randomly 
define 3D point Q’ lying on each of these light rays. 
Finally we use (2) to estimate the new value of The 
back ray tracing algorithm require to convert each 
mapping of the image formation model. However, the 
distortion function is not directly invertible. We used [5] 
to estimate the inverse distortion function. 

RESULTS 

We conduct experiments with three Storz fetoscopes 
model 26120a (33-degree angle of view), 27020aa and 
26003 aga (0-degree angle of view). Each dataset is 
composed of 15 dry and underwater calibration images 
as amniotic fluid is mainly made up of water. We 
compared underwater calibration results with the results 
obtained by estimating fluid-immersed camera 
parameters form a dry calibration. We used previously 
defined algorithm to compute We used a single 
underwater observation to compute y’ by minimising 
the reprojection error for the last mapping of the image 
formation process. We used underwater images to 


compare the two methods. Results are illustrated in 
figure 3 and synthesised in the following table. 


Fig. 3 Reprojection error. Red dots: ground truth, yellow and 
green circles: 3D points projected in the images using 
underwater calibration parameters and estimated parameters. 
Left: 33 degree fetoscope. Right: 0 degree fetoscope. 


Reprojection 

error 

(pixel) 

Fetoscope 1 
(0 degree) 

Fetoscope 2 
(33 degree) 

Fetoscope 3 
(0 degree) 

underwater 

calibration 

0.46 + 0.06 

0.43 + 0.43 

0.52 + 0.21 

Estimated 

calibration 

0.69 + 0.14 

0.70 + 0.12 

0.75+0.12 


The results show that estimated calibration is less 
accurate than a proper underwater calibration but the 
reproduction error is less than a pixel for all the 
fetoscope used for experiments. As most of the 
distortion effects are corrected underwater, a proper 
calibration of the fetoscopes using a central catadioptric 
model was difficult to obtain [3]. 

DISCUSSION 

We presented promising results which show that fluid- 
immersed endoscope parameters could be directly 
inferred from a dry calibration using the unifying 
catadioptric model. Investigations will be conducted in 
order to establish a proper link between £, and y. This 
could lead to a fast calibration process adapted to the 
constraints of fluid immersed endoscopy. 
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INTRODUCTION 

Minimally invasive surgical techniques have led to 
novel approaches such as Single Incision Laparoscopic 
Surgery (SILS), which allows the reduction of post- 
operative infections and patient recovery time, 
improving surgical outcomes. However, the new 
techniques pose also new challenges to surgeons: during 
SILS, visualization of the surgical field is limited by the 
endoscope field of view, and the access to the target 
area is limited by the fact that instruments have to be 
inserted through a single port. 

In this context, intra-operative navigation and 
augmented reality based on pre-operative images have 
the potential to enhance SILS procedures by providing 
the information necessary to increase the intervention 
accuracy and safety [1]. Problems arise when structures 
of interest change their pose or deform with respect to 
pre-operative planning, as usually happens in soft tissue 
abdominal surgery. This requires online estimation of 
the deformations to correct the pre-operative plan, 
which can be done, for example, through methods of 
depth estimation from stereo endoscopic images (3D 
reconstruction). The denser the reconstruction, the more 
accurate the deformation identification can be. 

This work presents an algorithm for 3D reconstruction 
of soft tissue, focusing on the refinement of the disparity 
map in order to obtain an accurate and dense point map. 
This algorithm is part of an assistive system for intra- 
operative guidance and safety supervision for robotic 
abdominal SILS [2]. 

MATERIALS AND METHODS 

3D shape reconstruction using stereo-images is a 
process composed of two main steps: the disparity 
estimation and the stereo-triangulation. Using the left 
and right images ( im i} im r ) captured by a stereo 
camera, the disparity is estimated searching 
corresponding points between images. These are the 
projection of the same world point into two image 
planes. The relative distance between the corresponding 
pixels is described by the disparity map. Stereo- 
triangulation consists in exploiting the detected 
correspondence and the geometry of the cameras to 
extract the 3D measurement of the observed scene [3]. 

In this section, the steps of an improved algorithm for 
surface reconstruction are described (Fig. 1). 


Disparity Estimation 

Images are rectified in order to simplify the stereo- 
correspondence analysis. An intensity pixel-based 
algorithm is used for local similarity searching. A sparse 
modified census transform [4] is applied to irrii and 
im r before matching. This method is robust against 
non-stationary exposure and illumination variations. It 
converts each pixel inside a moving window (nxn) into 
a bit vector, representing which neighbor pixels have an 
intensity above or below the central pixel and the mean 
of the pixels inside the window. 

The similarity between the two census transformed 
images is done using the sum of Hamming Distance 
(HD) measure applied on a moving window. HD 
compares bit strings representing the pixels and 
identifies the number of positions at which the 
corresponding bits are different. These values are 
summed within a window ( mxm ) and a “winner-takes - 
all” strategy is used to find the minimum, and thus the 
disparity associated to the best similarity. The HD is 
computed only for a chessboard pattern of pixels inside 
the window in order to decrease the computational time. 
Since using only pixel precision the reconstructed 
surface would consist of separated layers, a sub pixel 
refinement is applied using a parabola fitting. To 
invalidate wrong pixels on texture-less surfaces, the 
disparity is defined invalid if the two minimum values 
of HD are within a threshold. Additionally, a Left-Right 
Consistency (LRC) check is performed to invalidate 
half-occluded pixels, i.e. object views in one image and 
not in the other. A speckle removal filter is also applied 
in order to invalidate regions of large and small 
disparities that can be generated near the boundaries of 
the objects. 

Dense Disparity Optimization 

The pixels of the disparity map defined invalid lead to 
holes in the point cloud, compromising the usefulness of 
the reconstructed surface. Here, we exploit the 
segmentation of the reference image imi to optimize the 
disparity map (Alg. 1). In literature, these techniques 
segment the images and then label each region with a 
disparity value, preserving the boundaries of the objects. 
However, finding gradually changing disparities can be 
a problem since all pixels in a region have the same 
disparity. Moreover, segmentation algorithms are 
computationally expensive. 
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Algorithm 1 Label-based Disparity Optimization 
Require: D disparity map 

L t labeling of irrii segmented with SLIC Superpixel 
M mask of invalid disparity values 
x threshold of number of valid points 

1: associate L t to D -» DL t 

for each DL t i e {1, . . nSuperPixel } 

3: compute plane parameters p with LO-RANSAC of 
4: DLfi,j ) with i = n cols , j — Throws 
5: compute median m of DLfi, j) 
end for 

for each i and j of D 
if M(i,j) = true 
if ( nValidPoints e DL i)>* 

6: replace D(i,j) with m ( DL t ) 

else 

7: replace D(i,j ) with a point on p(DL{) 

end if 
end if 
end for 

return disparity map D re fined 


accuracy and the computational time, as shown in Fig. 
2. We considered also the percentage of valid points to 
evaluate the density of the point cloud (Fig. 2 - black 
line). Results show how the smoothing improves the 
accuracy of the reconstruction, giving a median error 
between 1.50mm and 2.27mm in the worst case. 
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Fig. 2 Error before (green line) and after (blue line) the 
smoothing, and the pre-smoothing computational time (red 
line) with the corresponding interquartile range. CnHm 
represents the dimension of a squared window used in census 
transform (C) and in HD (H), ordered increasingly with time. 


To overcome these drawbacks, imi is segmented using 
a Simple Linear Iterative Clustering (SLIC) super pixel 
algorithm [5]. Each super-pixel is an homogeneous area 
with similar or at least continuous depth. A label L is 
assigned to each super-pixel, being im^L) a group of 
pixels with the same label, i.e. belonging to the same 
super-pixel. Only the invalid values of disparity map are 
corrected using the information of the pixels belonging 
to the same label. Depending on the valid values inside 
a label, we apply two different strategies: 

■ Plane fitting: if the found plane can be considered a 
reliable plane, the invalid disparity values are fitted 
to that plane. The plane parameters are estimated 
using the Locally Optimized RANSAC method. 

■ Constant fitting : otherwise, the invalid disparity 
values are replaced by the median of the valid ones. 

After the stereo -triangulation step, a Moving Least 
Square algorithm is applied in order to smooth the point 
cloud, using PCL library. 


Fig. 1 Main steps of the presented algorithm. From left to 
right: RGB image, census transformed image, disparity map, 
dense refined disparity map. 

RESULTS 

The point clouds obtained are validated using 10 frames 
from a video dataset ( heart 1) available online from the 
Hamlyn Centre [6], the closest available datasets to the 
application of this work. The error is defined as the 
median of Euclidean distances between the point cloud 
and the ground truth. We calculated this error for 
different window sizes of census transform and 
Hamming Distance to show the correlation between the 


DISCUSSION 

The presented work provides a dense surface 
reconstruction to be used in an enhanced vision system 
for robotic abdominal SILS. The strength of this method 
is the usage of SLIC Super Pixel algorithm for obtaining 
a high density valid disparity map (over the 90% of the 
total number of points), which can be exploited for 
augmented reality applications. Comparing our method 
with state-of-the-art CPU implementations [7] evaluated 
on heart 1 , the percentage of valid points obtained with 
our method is 40% higher while providing a slightly 
lower accuracy (mean error 2.38 mm). Future research 
will focus on the development of a real-time 
implementation of the proposed algorithm, potentially 
based on a hybrid CPU-GPU processing framework. 
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INTRODUCTION 

Active Constraints (ACs) are high-level control 
algorithms deployed to assist a human operator in man- 
machine cooperative tasks [1], and define regions within 
which it is safe for the robot to move and cut [2]. To 
enhance the performance in cooperative surgical tasks, 
adaptive constraints have been exploited to optimally 
adjust the provided level of assistance according to 
some knowledge of the task, hardware or user. In [3] 
Hidden Markov Models were used for the run-time 
detection of the user intention to leave a guidance 
constraint to circumvent an obstacle. In this work, we 
present a novel, Neural Network (NN)-based method for 
the runtime classification of intentional and 
unintentional violations of ACs, that is trained on either 
statistical or frequency features from the enforced 
constraint forces. We investigate which set of 
parameters yield faster and more reliable classification 
results, both for guidance and regional constraints. 

METHODS 

Active Constraints 

During cooperative assistance, intentional violations of 
ACs take place whenever the current action of the user 
is in disagreement with the purpose of the constraint, 
typically resulting from sensing limitations of the 
robotic system. In this case, the constraint is felt as a 
hindrance, resulting in disturbing interaction forces at 
the tip. Unintentional violations occur when the user 
shares the purpose of the constraint and accidental 
errors in the task execution are made. The classification 
of the user’s intended action during the cooperative task 
would allow one to optimally adjust the assistance level 
provided by ACs. ACs can have two purposes [1]: 

Guidance constraints are enforced to guide the 
motion of the tool along a specified trajectory; 
Regional constraints are enforced to bound the 
motion of the tool into certain safe regions. 

Both types of constraints were considered in this work, 
and modeled with a planar geometry according to a 
conventional viscoelastic constraint model: 

f = K{x - x eq ) - Dx (1) 

where x and x are the position and velocity of the tool 
tip; k and D are the stiffness and damping parameters 


(K > 0, attractive fixture), x eq is the equilibrium point 

that lies on the constraint where it is closest (Euclidian 
distance) to the current tool tip position. To enforce the 
constraints during the cooperative guidance, the 
commanded torque of the haptic master is computed 
from the resulting Cartesian force / according to the 
geometrical Jacobian. 

Classification Method 

Two NN-based binary classifiers were developed for the 
runtime identification of “intentional” and 
“unintentional” violations of ACs. The two approaches 
exploited different features, extracted from the 
interaction force signal across the tip-constraint 
interface, as follows: 

1. Statistical (StNN): A feedforward NN was 
trained and validated on a dataset of 7 
statistical features computed on the temporal 
evolution of the interaction force: mean, 
variance, energy, maximum value, integral, 
waveform length, average amplitude change; 

2. Spectral (SpNN): A feedforward NN was 
trained and validated on a dataset of 10 spectral 
features computed on the time evolution of the 
force energy distribution using the Wavelet 
decomposition (9 levels) [4] ; 

The structures of the StNN and SpNN were composed 
of one hidden layer (15 and 33 neurons respectively, 
“trial and error” optimization), and one output neuron 
(hyperbolic tangent activation functions). 

EXPERIMENTAL DESIGN 

Experimental setup 

The Phantom Omni (Sensable Technologies, Inc.) 
haptic device was used during assisted cooperative 
tasks. The active constraint controller was implemented 
using the “PhanTorque” 1 Simulink-compatible library 
on Matlab/Simulink R2014b platform. Visual feedback 
of the task execution was provided with a 2-D monitor. 


1 https://sir.upc.edu/wikis/roblab/index.php/Projects/Pha 
nT orqueLibraries 
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Experimental protocol 

To evaluate task-independent properties of the two 
classifiers, two sets of tasks were considered (Fig. 1): 

Following task. The user was asked to 
accurately move along 2D spline -based paths, 
assisted by a guidance constraint. He/she was 
asked to circumvent any circular obstacles 
placed along the path by acting against the 
constraint; 

Reaching task. The user had to accurately place 
the pointer on several equally spaced targets, 
which lay within a forbidden region bounded 
by a regional constraint with 50% probability. 




Fig. 1. Following Task (left) and Reaching Task (right) 

We asked 12 subjects to perform 10 trials for each of 
the two tasks. The force signal was filtered, recorded, 
segmented to extract non-null interactions ( f > 0 ) and 
labeled as intentional or unintentional according to the 
known positions of obstacles/region boundaries. Two 
task-related datasets were built from all users across all 
trials. Both SpNN and StNN methods were cross- 
validated on each task dataset, and the performance 
were computed in terms of sensitivity (Se) and 
specificity (Sp) over time as: 

Se(t) = TP(t) /(TP(t ) + FN(t)) (2) 

Spit) = TN(t)/(TN(t) + FP(t)) (3) 

where T P and T N are the amount of correctly classified 
intentional (true positive) and unintentional violations 
(true negative), respectively; FN and FP are the 
amount of misclassified intentional (false negative) and 
unintentional violations (false positive), respectively. 

RESULTS AND DISCUSSION 

The classification threshold applied on the continuous 
output of the networks was optimized to achieve 90% 
asymptotic specificity. Consequently, the classifier was 
evaluated in terms of sensitivity- time profile (Fig. 2), 
and the minimum time interval to overcome a 90% 
sensitivity level was obtained. The Se index shows a 
sigmoidal profile in time for both tasks and methods. As 
reported in Table I, the 90% sensitivity was achieved for 
both methods within Is for the following task (mean 
velocity 4.17 mm/s), and within 3s for the reaching task 
(mean velocity 8.36 mm/s). In the following tasks, as no 
motion limitation was imposed to the user during 
obstacle avoidance, a greater penetration was recorded 


with respect to the reaching task. Moreover, the StNN 
method resulted in higher performance with respect to 
SpNN method, yielding a 60% time reduction for the 
following task, and a 30% reduction for the reaching 
task. The proper classification timing is chosen based on 
the sensitivity level required by the specific application. 



Time [s] 

Fig. 2. Sensitivity profiles in time, across methods and tasks. 


Table I Classification time and relative constraint penetration 
among methods and tasks (Se = 90%). 


Task 

Method 

Time 

[s] 

Penetration 

[mm] 

Following 

StNN 

0.559 

0.82 

SpNN 

0.954 

1.21 

Reaching 

StNN 

0.816 

0.51 

SpNN 

2.911 

0.52 


CONCLUSIONS 


NN-based algorithms were demonstrated to be suitable 
for the runtime task-independent classification of 
intentional and unintentional violations of ACs. Better 
performance was obtained for the guidance constraints 
with respect to regional constraints. Additionally, the 
use of statistical features yields a faster classification 
with respect to the use of spectral parameters. Future 
work could apply multi-objective model selection to 
find the optimal classifier, introduce data regularization 
(subsampling) to prevent class unbalance and combine 
statistical and spectral features. Moreover, some 
methods to exploit the continuous output of the NN 
could be investigated to optimally modulate the 
constraint assistance according to the probability of the 
user’s intention classification in surgical manipulation 
tasks. 
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INTRODUCTION 

Minimally invasive surgery first and, more recently, 
surgical robotics have brought new perspectives to 
surgery and have significantly improved the quality of 
many critical surgical tasks [1,2]. 

In this paper we go further by describing the design and 
testing of an architecture to control a robotic platform 
capable of executing in an autonomous way basic 
surgical tasks such as US -guided needle insertion and 
suturing. The results have been obtained during the FP7 
European funded project Intelligent Surgical Robotics 
(I-SUR, grant agreement n. 270396). The main goal of 
the project was to demonstrate that autonomous surgical 
robotics can carry out simple surgical tasks effectively 
and without major intervention by surgeons. To fulfil 
this goal, we have developed an innovative approach 
(both in terms of technologies and algorithms) for 
planning and execution of movement of robot arms in 
contact with the deformable anatomical environment. 

In [3,4] we described a methodology to collect 
requirements from surgeons, to translate the 
requirements in a formal and goal-oriented language, 
and to design sequence and state diagrams in Universal 
Model Language (UML) that can be implemented by an 
autonomous robotic platform. The trajectory planning 
based on pre-operative data (CT or MRI) is performed 
automatically as reported in [5]. The control and 
software architecture is described in [6]. Since the 
architecture of the system is modular and component- 
based by design, the same methodology can be applied 
to automate both the puncturing and suturing tasks. 

With the proposed approach, the robotic system 
autonomously deals with the simplest tasks of a surgical 
procedure, whereas the surgeon can focus his/her 
attention on the most critical parts. The surgeon 
monitors the automatic execution of the task thanks to a 
user interface where all the relevant data of the 
procedure are clearly shown, and warning notices are 
displayed. 


CASE STUDY AND ROBOTIC SETUP 

To develop the system requirements we focus on 
surgical actions that would benefit mostly from the 
accuracy and repeatability levels achievable by 
intelligent robots. We selected percutaneous 
cryoablation of small tumors and automated suture of 
planar wounds as our case studies. Percutaneous 
cryoablation requires the use of pre- and intraoperative 
images (CT, MRI/US) to insert, through the skin, one or 
more cryoprobe needles into the tumoral mass to be 
destroyed and to check the real-time position of the 
tools inside the patient. Trajectory misalignments are 
usually due to the deformation of soft tissues and organ 
displacement because of breathing. Thanks to real-time 
image registration and accurately calibrated mechanical 
arms, needle insertion will be executed precisely by the 
robots and monitored via US images as described in [7]. 



Fig. 1 Experimental setups for the needle insertion (Left) and 
for the suturing (Right). 


To evaluate practical issues and benefits of cryoablation 
execution by means of automated robots, an 
experimental setup was prepared, as shown in Figure 
l(Left). The setup includes two robotic manipulators: 
the ISUR micro-macro robot (developed by ETH) holds 
the needle, while the commercial UR5 robot holds the 
US probe. The end-effectors of both robots are equipped 
with specific tool adapters that integrate 6-DOF 
force/torque sensor. We used an Accutrack 500 from 
Atracsys LLC, Switzerland, to estimate the relative 
coordinate transformations among the robots, the 
phantom and the US images. Fig. 1 (Right) shows the 
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setup for the suturing: the ISUR robot is equipped with 
two arms: the left arm holds the Covidien Endostich for 
performing the suture whereas the right arm is needed to 
grasp the thread to avoid knots during the procedure. In 
the experimental section we will focus on puncturing 
due to the space limitation and provide web link for 
videos related to suturing. 

RESULTS 

In this section the time series of forces and the states of 
the robots control logic recorded during the 
experimental phase are reported (see Figures 2 and 3). 
The proposed experimental setup goes through the 
following steps triggered by the surgeon through the 
user interface: 

(1) The surgeon pushes the Ready button. Both robots 
move from their nest position to the ready position 
where the needle and the US probe are mounted. 

(2) The robots receive their nominal trajectories and 
move the needle and the US probe in contact with the 
phantom. 

(3) When the surgeon pushes the InsertNeedle button, 
the robot holding the needle starts the insertion until it 
reaches the target point as monitored by the US probe. 
In this case the state of the control logic is 
“WaitCryoCycle”. However if the surgeon decides to 
stop the procedure ( Stop button) because the needle is 
not visible in the US image any more, the robot holding 
the US probe performs a sequence of movements to 
bring the needle back on the US images. 

(4) The surgeon pushes the button Finish to bring the 
robots back in the “Ready” state. 

Figure 3 (Top) shows the force measured by the F/T 
sensor located on the robot holding the needle along the 
main axis of the needle. The dashed vertical lines and 
the numbers help to understand what state of the control 
logic (Figure 2) is activated. Figure 3(Bottom) reports 
the Cartesian tracking error w.r.t. the nominal trajectory. 
At the following link a video of the US -based needle 
insertion can be seen: 
http://ldrv.ms/lF4z6v4 

whereas the links: 
http://ldrv.ms/lF4zaLo 

http://ldrv.ms/lF4zbvN 

refer to the on-going development of the suturing task. 

DISCUSSION 

In this paper we presented a robot control and 
coordination framework for the automation of simple 
surgical tasks. We formalized the design specifications 
using a requirements engineering approach and derived 
the state machines for the control of the robots. Then, 
we implemented the proposed architecture using 
component-based design tools, i.e. Orocos framework 
and rFSM, in order to properly handle the distributed 
nature of our system. The goal of these experiments was 
to show that the technology to execute autonomously 
basic surgical tasks is available and could be used to 
improve accuracy as requested by surgeons and 
radiologists involved in the ISUR project. 



1 :lnit Robot 



Fig. 2 UML State Diagram for the needle insertion task. 



Fig. 3 Top. Force applied to the needle during the emulated 
cryoablation task; Bottom: tracking error (position) during the 
approach and insertion phase. 
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INTRODUCTION 

The rigid structure of laparoscopic instruments 
restricts manoeuvrability and reduces the possibility to 
reach surgical targets that are not situated along the line- 
of-sight of the entry point. In such a situation, the 
surgeon needs to reposition the laparoscopic instrument 
with the risk of damaging healthy tissue and organs. 

In order to address this limitation, several articulated 
surgical instruments have been developed. Prominent 
examples of such flexible systems include the i-Snake 
[1], HARP [2], active-cannula based systems [3], and the 
STIFF-FLOP arm [4]. These robots often consist of 
curved segments joined serially, which enable a high 
degree of articulation and increased manoeuvrability in 
confined spaces such as the human body. It is noted 
though, that the control of these robots requires precise 
shape and position sensing. The purpose of this research 
is to achieve shape sensing through multi-segment 
curvature sensing. Review of similar studies reveals that 
the motion of these robots have been mainly tracked 
using vision [5] or electromagnetic tracking [6]. 
However, vision based systems are often restricted due 
to visual occlusion, and electromagnetic tracking is 
subject to magnetic field distortions, in addition to its 
spatial range being limited to the generated magnetic 
field [7]. Fibre Bragg Grating (FBG) sensors have been 
proposed for shape sensing in continuum robots [7]; 
nevertheless, these systems are highly sensitive to the 
change in environmental temperature. Also when 
subjected to a non-uniform strain field, the strain 
compensation becomes complex [8] . 

This paper presents a highly compact system 
composed of optical fibre bundles for multi-segment 
shape sensing based on light intensity modulation, with 
the capability of modular sensing. 

MATERIALS AND METHODS 

The proposed arrangement is 7.5 mm in diameter and 
comprises a flexible and stretchable Steiner chain section 
[9] with twelve housings at the tip which accommodates 
driving tendons and passive cables used for sensing. The 
Steiner chain section is followed by a basal rigid optical 
fibre sensing unit (Fig. 1), where the passive cables are 
mechanically coupled with optical fibres. The optical 
fibre sensing unit is integrated with a low-friction 
retractable distance modulation array (Fig. 2a). This 
mechanism uses a steel needle-spring double slider (Fig. 
2c) connected to the base through an extension spring 
(Fig. 2e) to reduce the hysteresis and recover reference 
sensor values when the arm returns to its original straight 
configuration. In order to place the emitter and detector 
next to each other, the fibre was looped (Fig. 2d). 


Through the loopback design of the sensing system, all 
electronics are kept away from the sensing site. 



distance 
- modulation 
array 


Steiner chain 
section 


Fig. 1. The overall structure of the shape sensing system. Note 
that the flexible section also provides an empty central channel 
for end-effector instruments (CCS1 and CCS2 indicate 
constant curvature segments with radii of r! and r 2 , 
respectively). 



steel needle- spring loopback / 

double slider optical fiber extension spring 


Fig. 2. The prototype of the arm and sensing system: (a) the 
finished configuration of the sensorized manipulation system, 
(b) close-up view of the top part, (c) the low-friction steel 
spring- needle double sliding mechanism (d) the loopback 
optical fibre arrangement, and (e) the recovery mechanism of 
the slider using an extension spring. 
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The curvature sensor of each segment of the arm 
consists of three passive cables. These cables are sliding 
inside flexible Steiner chain housings in the periphery of 
the arm, as shown in Fig.l, in parallel with driving 
tendons. When the arm bends, the cables’ length portion 
inside the arm (si, s 2 and s 3 ) will change according to the 
pose of the arm. The changes in the length of these 
equally spaced cables modulate the position of the 
transmitting optical fibres (emitters) and, consequently, 
the intensity of light received by detectors (e.g. FS- 
N 11 MN, Keyence™). The received light is then 
converted into voltage v = [vi v 2 v 3 ]. By relating the 
voltage to the corresponding unique distance vector s = 
fly) = [si s 2 S3], the configuration parameters of the arm 
are acquired for each specific pose, which are computed 
as follows [10]: 


5 = i£? = i(S;), e = 


S — 


d cos(— cp) 1 


f = K "' i c g 3fa-;r >) 


where 5 is the arc length of the central axis of the each 
segment, 6 is the bending angle, cp is the orientation 
angle, and d describes the distance between the central 
axis of the manipulator and the parallel sensing tendons 
(here d=2. 1 mm). 

This sensing arrangement uses only three Steiner 
channels out of twelve, extended along the whole length 
of the arm, for sensing. There are two passive cables 
sliding inside each of these three channels. One of the 
cables is fixed between two segments and the other one 
is fixed at the tip of the manipulator. This 
implementation allows measuring the bending angle in 
multi-segment arms in a modular way, in the sense that 
the Steiner chain structure remains the same for all 
segments. 

RESULTS 

In order to validate the design and implementation of 
the two-segment shape sensor, we have performed a set 
of experiments and measured the amount of bending 
curvature in each segment (Fig. 3 ). Two cameras were 
placed at the top and side of the manipulator to record 
ground-truth shape information. 

DISCUSSION 

This paper presents a novel shape sensing system for 
a multi- segment flexible manipulation arm. Three 
Steiner Chain channels were used for shape sensing in a 
two-segment articulated arm. A low-friction optical fibre 
distance modulation array based on a new spring-needle 
double slider was designed to precisely modulate the 
distance between emitter and detector optical fibres 
during the arm’s bending. The design also employs a 
loopback optical method to keep all electronics away 
from of the sensing site. The complete sensing system is 
implemented and experimentally evaluated. A maximum 
error of 6° with respect to the camera ground-truth 
information was determined. From the experimental 
results, it can be seen that the sensing error can be 


reduced by increasing the initial stretch length of 
extension springs, hence, tightening the passive cables 
used for sensing. 



• base segment (curvature sensor) 

• base segment (camera ground truth) 

■ tip segment (camera ground truth) 

— ■ — tip segment (curvature sensor) 

Fig. 3. The experimental results of curvature sensing in 

individual segments. 
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INTRODUCTION 

The advent of surgical robot technologies has generated 
considerable impact on minimally invasive surgery. 
Equipped with multi-axis force sensors, surgical robots 
will endow surgeons with the essential feedback of 
touch to ensure the efficiency and safety of operations. 
Patients can also be protected from potentially excessive 
forces applied onto delicate internal organs. There are 
various miniaturized multi-axis force sensors that could 
potentially be considered for integrating with surgical 
tools, including those based on piezoresistive materials, 
strain gauges, polyvinylidene fluoride films, and fibre- 
optics. Commonly, two methods: (1) light intensity 
modulation or (2) Fibre Bragg Grating (FBG) have been 
utilized for developing fibre optic sensors [1-4]. In this 
paper, we propose a novel force sensing approach, 
making use of fibre optic technology and a CCD 
camera, which acquires light intensities transmitted 
from fibres. The light intensities are then modulated as a 



Fig. 1 The sensor mechanical structure based on the three 
cantilever beams 



Sensor structure 

Mirror 


Reflective light 


Multi-core optical fibre bundle 

One emitting fibre 
Three receiving fibres 


FS-N11MN Fiberoptic 
by Keyence Co v Ltd. 


One optical fibre 
(emitting source) 


Fig. 2 The configuration of the sensor principle overview 
for multi-axis force sensor 


function of applied forces. As the cost of CCD chips has 
been dramatically reduced, this new approach provides 
a cost effective solution for fibre-optic sensing, 
especially when a large number of fibres are required. In 
this paper, we present the design and development of a 
three-axis force sensor using the proposed approach. 
This paper introduces the design, fabrication and 
calibration procedures. The experimental results show 
that the developed sensor achieved good performance. 

MATERIALS AND METHODS 

Sensor Principle: The proposed multi-axis force 
sensor consists of a CCD camera, a multi-core optical 
fibre bundle (FU-49X, Keyence Corp., Japan) and a 
light source. A light source (FS-N11MN, Keyence 
Corp., Japan) was used in this study. The sensor 
structure is designed to measure three force components 
F x , F y , and F z ; the three force components can be 
calculated by measuring the deflections Si, S 2 and S 3 of 
the three cantilever beams, Fig. 1. 

The force measuring principle of our sensor is shown as 
follows. Light emitted from the light source travels via 
the emitting fibre; the light is then reflected by mirrors 
attached to the three cantilevers which bend in response 
to an applied force; the reflected light is then 
transmitted via three corresponding receiving fibres to 
three separate regions of the CCD camera; the light 
intensities in three sub regions of the CDD are 
interpreted into three corresponding cantilever beam 
deflections S u S 2 and S 3 and the associated forces, Figs. 
1 and 2. 


Image Processing: The image with three sub-regions 
of reflected light is shown in Fig. 3a. To determine the 
light intensity of the three regions, signal processing 
approach first applied a median smoothing filter to the 
original RGB image reducing the original image noise. 
Then the image is converted into a greyscale image, Fig. 
3b. To calculate each of the light intensity values of the 
three receiving fibres, the three corresponding regions 
of interest (ROIs) on the greyscale image are further 
processed. Each ROI has a pixel size of 320x240. The 
values of all pixels within the ROI are summed and 
divided by normalizer to obtain the reflected light 
intensity. From these normalised light intensity values 
(dj, a 2 , and d 3 ), the three cantilever beam deflections 
and the associated force components can be calculated, 
Fig. 1. To further reduce the noise, filtering techniques 
such as Kalman filter could be applied. 
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Fig. 3 Original and processed images, the captured by the 
camera, from three light channels: a) the original RGB image 
b) the greyscale image converted after applying median 
smoothing filter on the original RGB image. 


RESULTS 

A set of the accuracy tests have been carried out with 
the multi-axis force sensor integrated into the catheter 
tip to compare the estimated forces with benchmark 
values (load cell) (Figure 5 left bottom). The results 
show that the estimated force components F z , F x and F y 
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Fig. 4. Verification of sensor performance test: a) 
comparison between real F z and estimated F z b) 
comparison between real F x and estimated F x (cross-talk in 
other force components F y and F z ) c) comparison between 
real F y and estimated F y (cross-talk in other force 
components and FV). 



(a) (b) 


Fig. 5 Possible medical devices for image based optical 
multi-axis force sensor: a) palpation tool (left top), da 
Vinci SutureCutTM Needle Drivers (left middle), and 
catheter tip (left bottom), b) flexible continuum 
manipulator. 

closely align with the benchmark forces, Fig. 4. It can 
be noticed that F y has slightly higher crosstalk when 
compared to the estimations of F z and F x , and has a 
higher error in comparison with F x and F y . This could 
possibly be due to an asymmetrical arrangement of the 
receiving fibres as shown in Fig. 2. When the maximum 
force is applied to the three-axis force sensor, the 
maximum error is 34% ( F x ), 41% (F y ) and 13% (F z ), 
respectively. 

DISCUSSION 

The estimation of the force components calculated by the 
decoupling stiffness matrix has errors in comparison 
with the benchmark forces. We think these errors may be 
caused by the configuration of the asymmetrical optical 
fibre arrangement. As a future task, we will propose a 
symmetrical optical fibre arrangement (an optical fibre is 
located in the centre, and three are symmetrically located 
at 120° at the same radial distance) to reduce the 
observed estimation errors, and also further investigate 
sensor characteristics such as nonlinearity, crosstalk, 
repeatability and hysteresis. Furthermore, our proposed 
multi-axis force sensor will be integrated into many 
medical devices as shown in Fig. 5. 
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INTRODUCTION 

Developments in the field of epilepsy surgery in the last 
25 years have all been technology driven. MRI imaging 
of mesial sclerosis and the minimally invasive surgical 
techniques enhanced by the advent of image guided 
surgery have made temporal lobe surgery a routine 
procedure with greater than 70% cure rates in most 
centres. However imaging negative temporal lobe 
epilepsy and epilepsy of extratemporal origin remain a 
difficult challenge surgically. Historically, investigation 
techniques using subdural grids were imprecise, being a 
two dimensional solution to a three dimensional 
problem and carried unacceptably high risk for an 
investigative procedure [1]. The introduction of 
neurosurgical robotics and the order of magnitude 
greater accuracy that the technology provides has 
provided a solution to the investigation of extratemporal 
epilepsy. 

RESULTS 

This paper is an observational study comparng the 
results of conventional subdural grid placement for 
Video Electrocorticography (ECoG) recording with 
Robotic Stereo Electroencephalography (SEEG). 

Between 2001 and 2011, 30 ECoG cases using subdural 
grids were performed out of a total of 149 craniotomies 
for epilepsy (20%) ECoG localised seizure onset 
sufficient to allow resective surgery in 27 cases (90%), 
A temporal focus was defined in 12 cases and an extra 
temporal focus defined in 15. Surgery lead to cure of 
epilepsy in 17 cases (63%), 9/12 temporal lobe cases 
(75%) and 8/15 extratemporal cases (53%) However 
seven patients suffered major complications from 
surgery - 4 haematomas, 2 infections and 1 stroke 
(23%) limiting the application of this technique as an 
investigative procedure. 

Since 2011 ECoG has been replaced with robotic 
SEEG, carrying out 32 cases out of a total of 80 
craniotomies for epilepsy (40%), In 17 cases SEEG was 
used to evaluate complex temporal epilepsy and 
extratemporal epilepsy in 14 cases. SEEG evaluation led 
to resective surgery in 20 cases with a cure of epilepsy 
in 16. (80%). Two patients are awaiting resective 
surgery. Seizures were shown not to be localizable in 9 
cases. In two, repeat SEEG will be necessary as the 
epilepsy onset, although focal, was distant from the 
recording electrodes. A total of 402 electrodes - an 
average of 12 per case (range 6-17) have been 


implanted with one surgical complication, an 
intracerebral haematoma (3% of the SEEG cases) 

CONCLUSIONS 

The small numbers in this series, particularly of any sub 
group analysis do not allow for useful statistical analysis 
that might help to define the efficacy of SEEG over 
ECoG in the investigation of epilepsy. However the 
reduction in surgical complications allow us to conclude 
that robotic SEEG is a major technological advance in 
the investigation of intractable epilepsy, providing a 
definitive answer with respect to seizure localisation in 
29/31 cases (94%) with subsequent curative surgery in 
16/20 (80%) cases 

DISCUSSION 

Currently electrode placement is defined on an 
individual case basis by the preoperative hypothesis of 
epilepsy propagation. Analysis of the SEEG recording is 
qualitative. The field is ripe for the development of data 
mining algorithms using neuromorphometric methods to 
standardize electrode positioning to enhance automatic 
analysis of the onset and spread of complex epilepsy. 
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ABSTRACT 

A cost analysis has been carried out for more than 700 
consecutive patients across therapeutic boarders in two 
Norwegian hospitals. Short and long term costs are 
calculated using a consistent homogeneous 
methodology. There are differences in costs between the 
management options as well as between patients within 
the same option. The cost information is an important 
prerequisite in cost-effectiveness analysis. 

BACKGROUND 

Non-Metastatic Prostate Cancer (n-MPCa) is the most 
common cancer in men. Radical prostatectomy (RP), 
Radiation therapy (RT) are standard curative approaches 
and Active Surveillance (AS) is increasingly accepted 
as a management option for localised n-MPCa. 
Comparative cost studies across therapeutic boarders are 
lacking. Methodological heterogeneity in existing cost 
studies is an obstacle for evidence synthesis as input to 
comparative HTA studies [1,2]. 

OBJECTIVE 

To present costs components and cost structures for RP 
(Radical Retropubic Prostatectomy (RRP), Robot 
assisted Laparoscopic Prostatectomy (RALP)), RT (EB 
and HD-BRT) and AS as management options in n- 
MPCa with emphasis on initial costs and long term 
follow up costs. 

METHODS 

The study includes 711 consecutive patients diagnosed 
with n-MPCA managed with curative intent in 2011 - 
2013. All patients are referred from GP’s to two 
Norwegian hospitals. The cost data are derived from 
both administrative hospital records and electronic 
patient records data. The cost units were identified 
through literature search and a thorough patient care 
pathway analysis in close cooperation with clinicians. 
The follow-up costs include both hospital and GP visits. 
Costs are calculated from a health care provider 
perspective and presented in 2013 value of GB£. The 
study is carried out as a mixed micro -costing/case-mix 
method. 


RESULTS 

The immediate treatments (RP and RT) have high initial 
average costs per patient ranging from RRP at £4.304 
to, RALP at £ 6063, RT-HD-Brachytherapy at £ 9.875 
compared to AS at £119 for the first three months (see 
Figure 1). 

The cost of yearly follow-up is higher and has a 
different distribution in AS than RT and RP and RT. 
The cost of AS is highly dependent on the cross-over 
rate to active treatment. The follow up cost of active 
treatment is dependent on the adverse effects from the 
treatment and the incidence of salvage treatment (see 
Figure 2). 

The conversion rate from AS to active treatment plays a 
major role in this cost. In order to predict future cost 
from historic data a modelling process is required. In the 
present study the AS management option was the option 
accumulating the lowest costs over time followed by 
RRP, EB, RALP and HD-BRT. A transparent cost- 
model with natural cost units and unit costs can be 
adapted to local conditions and can therefore be 
generalizable with adaptation to local conditions and 
data. The results need to be seen in relation to 
oncological outcome and health state values. 

There are few cost-analyses across therapeutic 
boundaries. One single centre study from Miami, US, 
based on clinical data for individual patients, Medicare 
fees for physicians and average hospital costs for as unit 
costs. The results are similar in the ranking order of 
costs for different modalities and the cost structure. The 
difference was that RALP and HD-BRT were similar in 
cost while external beam was calculated as the most 
expensive option [3]. 

CONCLUSIONS 

In n-MPCa the main cost components and the cost 
structures in terms of onset of costs differ between the 
five management options included in the study. Active 
Surveillance incurs the least cost over time. The finding 
might be tested applying local results regarding 
treatment outcome, oncological outcome and adverse 
effects. 
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Cost of first three months of treatment 
by management option 
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Fig. 1 Cost per patient for the different management options 
for n-MPCa. The cost is for three months of treatment. The x- 
axis represent single patients and the n per modality. 
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Fig. 2 AS=Active Surveillance as observed, AS mod= 
Active Surveillance according to the latest protocol. 
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INTRODUCTION 

Brain tumors are one of the most feared complications 
of cancer and they can have a significant impact on the 
quality of life of an individual. Combining the intra- 
operative MRI system with an MRI-compatible robot 
with multiple degrees -of-freedom (DoFs) for higher 
dexterity, will enable us to develop a real-time MRI- 
guided robotic neurosurgical system that can provide 
neurosurgeons with the ability to remove the tumor 
more completely. Continuum robots have been 
extensively researched in robotics due to its flexibility 
and dexterity to work in confined spaces and also safely 
interact with its surrounding [1-3]. A soft continuum 
robot is an excellent choice of robot when a dexterous 
small-scale neurosurgical robot is needed to work in 
restricted space and avoid critical structures within the 
brain to prevent irreparable damage. We have developed 
a novel spring-based minimally invasive neurosurgical 
intracranial robot (MINIR-II), made of MRI-compatible 
plastic material. The robot body is a pair of parallel 
springs with an outer spring and an interconnected inner 
spring composed of multiple segments for better 
dexterity. 

In this continuum robot, the spring backbone is the most 
important part which provides the flexibility and rigidity 
as well as force transmission capability. We need to 
determine the appropriate stiffness and consider tip 
force transmission required for neurosurgery. Yoon and 
Yi [3] studied the stiffness model of a spring backbone 
and designed a compressible beam structure with the 
same stiffness as that of the spring, and estimated 
deflection based on the beam model. Actuation force, 
especially at the manipulator tip, plays a major role in 
allowing a desired robot movement in the brain tissue. 
It has been reported that a spherical surgical probe with 
a diameter of 2.5 mm has to overcome 0.1 -0.2 N of 
resistance force to advance through the brain tissue [4]. 
Our study on the tip force of the robot allows us to have 
a realistic expectation on the amount of tension we need 
to apply on the tendon to generate sufficient force for 
smooth robot motion in the brain environment. 

ROBOT DESIGN AND TENDON ROUTING 

Our robot has a spring that is divided into several 
segments (see Fig. 1) for the multiple DOFs needed in a 
neurosurgical procedure. It is made of a plastic material 
(VeroWhite, FullCure®, USA), commonly used in 3D 


printers. As shown in Figs. 1(a) and 1(b), together with 
the outer spring which acts as a flexible shell, the entire 
robot is assembled into a compact robotic device with 
the largest outer diameter being 13 mm. The inner 
spring is the main structure of the robot and at the same 
time offers flexibility and dexterity. The high DoFs of a 
flexible continuum robot make the tendon routing 
process complex. A slight movement of the tendons can 
easily deflect the robot links to a large extent when the 
tendons are routed using the 1 st configuration shown in 
Fig. 1(c). Decoupled joint motion is important so that 
actuation of one robot joint does not cause deflection of 
any proximal robot links. The 2 nd routing configuration 
shown in Fig. 1(c) leads to independent joint motion and 
therefore significantly improved robot performance. 
While tendon pulling generates moment at segment 3, it 
only causes normal compression in the other two 
segments. In this configuration, the tendons are routed 
along the central axis of the inner spring and start 
branching out to the respective robot joints at the base 
of the corresponding link. It is important to note that 
this design differs from the work of Bajo and Simaan 
[1], Jones and Walker [2], and Gravagne and Walker [5] 
because we can achieve independent control by routing 
the tendons central to the robot body and achieve 
motion in two independent directions for each link by 
using four tendons, spaced 90° apart. 

STIFFNESS MODELING AND FORCE 
ANALYSIS 



\ v 

Interconnected spring 


Seg ment 1 Segment 2 Segment 3 
I s1 configuration 
2 nd configuration 



Fig. 1 (a) MINIR-II robot (b) Spring design with outer spring 
and interconnected inner spring, (c) Two different tendon 
routing configurations (d) Design parameters of 
interconnected spring (e) Relationship between lateral 
deflection and moment arm. 
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During modeling of the stiffness of our spring-based 
robot, we utilized an actual spring model. Figure 1(d) 
shows a single round-wire spring with mean spring 
diameter, D, and wire diameter, d, loaded by a lateral tip 
force, F k . When force is applied at the tip of the spring 
to cause lateral deflection, there are two force 
components acting on it: direct shear force, which is 
equal to the tip force, F k> and torsion, T. The spring 
stiffness that relates the lateral deflection and the tip 
force, can be expressed as: 

k = — — (1) 

where N, H 2 and G are the number of spring coil, height 
of the tendon attachment point for tip force 
measurement and the shear modulus of the spring, 
respectively. The relationship between tip force, F k and 
tension, F T in the tendon can be expressed as: 

Z M = C F T ) y y ~ (F t ) x H 1 ~ F k H 2 = 0 (2) 

where Hj and y are the height and the lateral distance 
between the tendon attachment point and the center of 
rotation, respectively (See Fig 1(e)). To investigate the 
relationship between the spring deflection and different 
design parameters of the spring, a finite element (FE) 
model was developed. We assumed the material to be 
isotropic, homogeneous, and incompressible and we 
used ABAQUS (SIMULIA, USA) for our FE 
computation. The fixed boundary condition was applied 
on the bottom of the spring and a force in the lateral 
direction of 0.31 N was applied on the tip of the spring. 
For characterizing the spring constant, we performed a 
deflection experiment. The lateral motion at the end of a 
single robot segment was provided by a microstage 
(MP-285, Sutter instrument company, USA). While the 
end was moving laterally, the force exerted by the inner 
spring of the robot was captured by a force sensor 
(MDB-2.5, Transducer technology, USA). 




Wire diameter (mm) 



Mean spring diameter (mm) 

Fig. 2 Stress distribution of single segment of spring-based 
continuum robot and its tip deflection at various wire diameters and 
mean inner spring diameter. 


RESULTS AND DISCUSSION 

The simulation results from the FEM model (shown in 
Fig. 2) show that the deflection of the spring is 
nonlinearly dependent on the mean diameter of the inner 


spring and the wire diameter. From Eq. (1), stiffness of 
the inner spring, k, is 0.085 N/mm while it is 
determined to be 0.076 N/mm (R 2 =0.9814) through 
experimental data (see linear fit in Fig. 3). Compared to 
that of the inner spring, the stiffness of the outer spring, 
which is 0.009 N/mm, is negligible. It can also be seen 
from Fig. 4 that the tension applied is 4.4 times the 
exerted tip force. Therefore, considering safety factor, 
we need to ensure that our actuators are able to generate 
and withstand around 5 times the amount of force we 
need at the robot tip. The theoretical data from Eq. (2) 
matches the experimental data well (R 2 =0.9978). The 
robot can generate more than 1 N at the tip. In our 
future work, we will test the effect of stiffness changes 
on force transmission capability of our robot. 



Fig. 3 Relationship between the tension force on the tendon and 
lateral deflection. 



Fig. 4 Relationship between tip force and the tension force on the 
tendon. 
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